1043699 - Add Mulet support to b2gdesktop Reftest runner

Status: RESOLVED FIXED

(Testing :: Reftest, defect)

Description

[Jonathan Griffin (:jgriffin)]

We want to be able to run reftests on Mulet the same way as we do for b2gdesktop...i.e., in side the test-container gaia app.

The current b2gdesktop runner completely fails with Mulet at present.  It looks like there are several reasons for this:

- There are some gaia services that make non-local network connections, so MOZ_DISABLE_NONLOCAL_CONNECTIONS causes us to crash
- Launching the test-container app by setting b2g.system_startup_url fails; we're going to need to launch it via Marionette, which also probably means having to kill the FTU and lockscreen via Marionette
- Mulet doesn't recognize the --screen 800x1000 argument

Comment 1

[Jonathan Griffin (:jgriffin)]

Attachment: WIP - - Add Mulet support to b2g reftest runner

WIP, this allows the runner to at least launch Mulet with Gaia running in it.

Comment 2

[Jonathan Griffin (:jgriffin)]

I don't know how to cope with the fact that Mulet doesn't honor the --screen 800x1000 argument.  Alexandre, can we add support for this?  Reftests require this resolution and many will give false positives or negatives when run at the normal Gaia screensize.

Comment 3

[Alexandre Poirot [:ochameau]]

Attachment: Ensure that -chrome URI doesn't prevent window.arguments[0] to be nsICommandLine

The code to handle --screen is in Mulet, but the -chrome chrome://b2g/content/shell.html
arguments goes through another startup codepath which prevent window.arguments[0]
to be set to the nsICommandLine object! It ends up breaking b2g code implementing --screen argument.

Comment 4

[:Gavin Sharp [email: [e-mail]]]

Comment on attachment 8476005 [details] [diff] [review]
Ensure that -chrome URI doesn't prevent window.arguments[0] to be nsICommandLine

This almost certainly breaks Firefox, which doesn't expect its windows to receive a reference to the cmdLine object. If b2g expects that, it needs to adjust, or we need a special case.

Comment 5

[Alexandre Poirot [:ochameau]]

Attachment: Fix command line arguments handling on desktop

Given comment 4, it looks like window.arguments[0] being nsICommandLine
is an unexpected behavior...
So instead of fixing that to an expected behavior, I'm now working around this issue.

Comment 6

[Alexandre Poirot [:ochameau]]

https://tbpl.mozilla.org/?tree=Try&rev=6389ac4a840c

Comment 7

[Alexandre Poirot [:ochameau]]

Attachment: Fix command line arguments handling on desktop

Forgot to add CommandLine.js in package-manifest.in...

https://tbpl.mozilla.org/?tree=Try&rev=e45bd7303b25

Comment 8

[Alexandre Poirot [:ochameau]]

Try is finally green!

Comment 9

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/4d991522bbe3

Comment 10

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/4d991522bbe3

Comment 11

[Jonathan Griffin (:jgriffin)]

Forgot to 'leave-open'; there's still more work to be done.

Comment 12

[Armen [:armenzg]]

Attachment: mulet.diff

With jgriffin's patch we get it started, however, we have the lock screen.

Can marionette help me unlock the screen?

For reference, here's what it shows without the patch:
http://people.mozilla.org/~armenzg/sattap/86644eb8.png

wget http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-b1950fdbd196/try-linux64-mulet/firefox-35.0a1.en-US.linux-x86_64.tests.zip
wget http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-b1950fdbd196/try-linux64-mulet/firefox-35.0a1.en-US.linux-x86_64.tar.bz2
tar xvfj firefox-35.0a1.en-US.linux-x86_64.tar.bz2
virtualenv temp_venv
source temp_venv/bin/activate
pip install -e marionette/marionette
python reftest/runreftestb2g.py --desktop --profile=/home/armenzg/repos/branches/gaia/profile --appname=/home/armenzg/moz/tmp/mulet/firefox/firefox-bin --symbols-path=http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-b1950fdbd196/try-linux64-mulet/firefox-35.0a1.en-US.linux-x86_64.crashreporter-symbols.zip reftest/tests/layout/reftests/reftest.list

Comment 13

[Jonathan Griffin (:jgriffin)]

Yes, you can use Marionette to unlock the lock screen.

There is already a script executed via Marionette for B2G reftests:  http://mxr.mozilla.org/mozilla-central/source/layout/tools/reftest/b2g_start_script.js

I think you could adjust this to include code to unlock the lockscreen, namely the 'unlock' method from here:  https://github.com/mozilla-b2g/gaia/blob/master/tests/atoms/gaia_lock_screen.js

Comment 14

[Armen [:armenzg]]

Attachment: mulet.diff

I added gaia_lock_screen.js temporarily by b2g_desktop.py and I called it like this:
+        self.marionette.import_script(os.path.abspath(
+            os.path.join(__file__, os.path.pardir, "gaia_lock_screen.js")))
+        self.marionette.switch_to_frame()
+        self.marionette.execute_async_script('GaiaLockScreen.unlock()')

I was hoping to try to emulate this code:
https://github.com/mozilla-b2g/gaia/blob/master/tests/python/gaia-ui-tests/gaiatest/gaia_test.py#L682

There is also this that uses executeScript() instead
https://github.com/mozilla-b2g/gaia/blob/master/tests/js-marionette/helper.js#L7

My apologies it is clear on how to proceed. I've not deal much with JavaScript in the past. I should be able to make progress with some pointers.

However, I'm getting this:
1410556331093	Marionette	DEBUG	accepted connection on 127.0.0.1:44664
1410556336101	Marionette	DEBUG	accepted connection on 127.0.0.1:44665
1410556336127	Marionette	INFO	loaded marionette-listener.js
1410556336127	Marionette	INFO	sendToClient: {"from":"0","value":{"browserName":"B2G","browserVersion":"35.0a1","platformName":"LINUX","platformVersion":"35.0a1","handlesAlerts":false,"nativeEvents":false,"rotatable":true,"secureSsl":false,"takesElementScreenshot":true,"takesScreenshot":true,"platform":"LINUX","XULappId":"{ec8030f7-c20a-464f-9b0e-13a3a9e97384}","appBuildId":"20140911230945","device":"desktop","version":"35.0a1","b2g":true}}, {ec6b34c4-cc4a-4228-9fba-e05353186962}, {ec6b34c4-cc4a-4228-9fba-e05353186962}
1410556336130	Marionette	DEBUG	Got request: setContext, data: {"to":"0","sessionId":{"rotatable":true,"browserVersion":"35.0a1","takesScreenshot":true,"appBuildId":"20140911230945","XULappId":"{ec8030f7-c20a-464f-9b0e-13a3a9e97384}","secureSsl":false,"platform":"LINUX","browserName":"B2G","version":"35.0a1","device":"desktop","b2g":true,"nativeEvents":false,"platformVersion":"35.0a1","takesElementScreenshot":true,"platformName":"LINUX","handlesAlerts":false},"name":"setContext","parameters":{"value":"chrome"}}, id: {4e2aa646-d696-44dd-8f1c-091b9416e8e0}
1410556336130	Marionette	INFO	sendToClient: {"from":"0","ok":true}, {4e2aa646-d696-44dd-8f1c-091b9416e8e0}, {4e2aa646-d696-44dd-8f1c-091b9416e8e0}
1410556336132	Marionette	INFO	sendToClient: {"from":"0","ok":true}, {268a247d-8b5b-42d9-90f6-b13573756a84}, {268a247d-8b5b-42d9-90f6-b13573756a84}
1410556336133	Marionette	DEBUG	Got request: switchToFrame, data: {"to":"0","sessionId":{"rotatable":true,"browserVersion":"35.0a1","takesScreenshot":true,"appBuildId":"20140911230945","XULappId":"{ec8030f7-c20a-464f-9b0e-13a3a9e97384}","secureSsl":false,"platform":"LINUX","browserName":"B2G","version":"35.0a1","device":"desktop","b2g":true,"nativeEvents":false,"platformVersion":"35.0a1","takesElementScreenshot":true,"platformName":"LINUX","handlesAlerts":false},"name":"switchToFrame","parameters":{"id":null,"focus":true}}, id: {5ecb34f3-5a79-429e-b5ec-ac340cbebf07}
1410556336234	Marionette	INFO	sendToClient: {"from":"0","ok":true}, {5ecb34f3-5a79-429e-b5ec-ac340cbebf07}, {5ecb34f3-5a79-429e-b5ec-ac340cbebf07}
1410556336238	Marionette	DEBUG	Got request: executeWithCallback, data: {"to":"0","sessionId":{"rotatable":true,"browserVersion":"35.0a1","takesScreenshot":true,"appBuildId":"20140911230945","XULappId":"{ec8030f7-c20a-464f-9b0e-13a3a9e97384}","secureSsl":false,"platform":"LINUX","browserName":"B2G","version":"35.0a1","device":"desktop","b2g":true,"nativeEvents":false,"platformVersion":"35.0a1","takesElementScreenshot":true,"platformName":"LINUX","handlesAlerts":false},"name":"executeAsyncScript","parameters":{"scriptTimeout":null,"specialPowers":false,"script":"GaiaLockScreen.unlock()","newSandbox":true,"args":[],"filename":"b2g_desktop.py","line":44}}, id: {7585ad9a-0d1c-454c-9805-631b56ea66a0}
1410556336242	Marionette	INFO	sendToClient: {"from":"0","error":{"message":"TypeError: window.wrappedJSObject.SettingsListener.getSettingsLock is not a function","status":17,"stacktrace":"execute_async_script @b2g_desktop.py, line 44\ninline javascript, line 10\nsrc: \"undefined\""}}, {7585ad9a-0d1c-454c-9805-631b56ea66a0}, {7585ad9a-0d1c-454c-9805-631b56ea66a0}
PROCESS-CRASH | b2g_desktop.py | application crashed [unknown top frame]
Crash dump filename: /tmp/tmps6jzGQ/minidumps/1b6fce7d-f369-4d0f-22db530f-198a27dd.dmp
No symbols path given, can't process dump.
MINIDUMP_STACKWALK not set, can't process dump.
Traceback (most recent call last):
  File "reftest/runreftestb2g.py", line 611, in <module>
    sys.exit(main())
  File "reftest/runreftestb2g.py", line 606, in main
    return run_desktop_reftests(parser, options, args)
  File "/home/armenzg/moz/tmp/mulet/reftest/b2g_desktop.py", line 191, in run_desktop_reftests
    sys.exit(reftest.run_tests(args[0], options))
  File "/home/armenzg/moz/tmp/mulet/reftest/b2g_desktop.py", line 99, in run_tests
    self.run_marionette_script()
  File "/home/armenzg/moz/tmp/mulet/reftest/b2g_desktop.py", line 44, in run_marionette_script
    self.marionette.execute_async_script('GaiaLockScreen.unlock()')
  File "/home/armenzg/moz/tmp/mulet/marionette/marionette/marionette.py", line 1253, in execute_async_script
    filename=os.path.basename(frame[0]))
  File "/home/armenzg/moz/tmp/mulet/marionette/marionette/decorators.py", line 35, in _
    return func(*args, **kwargs)
  File "/home/armenzg/moz/tmp/mulet/marionette/marionette/marionette.py", line 638, in _send_message
    self._handle_error(response)
  File "/home/armenzg/moz/tmp/mulet/marionette/marionette/marionette.py", line 686, in _handle_error
    raise errors.JavascriptException(message=message, status=status, stacktrace=stacktrace)
marionette.errors.JavascriptException: JavascriptException: TypeError: window.wrappedJSObject.SettingsListener.getSettingsLock is not a function
stacktrace:
	execute_async_script @b2g_desktop.py, line 44
	inline javascript, line 10
	src: "undefined"

Comment 15

[Jonathan Griffin (:jgriffin)]

Comment on attachment 8488849 [details] [diff] [review]
mulet.diff

Review of attachment 8488849 [details] [diff] [review]:
-----------------------------------------------------------------

I think this is very close, just a chrome/content mismatch here!  Nice work!

::: ../mulet.original/reftest/b2g_desktop.py
@@ +36,5 @@
>      def run_marionette_script(self):
>          self.marionette = Marionette(**self.marionette_args)
>          assert(self.marionette.wait_for_port())
>          self.marionette.start_session()
>          self.marionette.set_context(self.marionette.CONTEXT_CHROME)

The lockscreen is accessible only in content context, so you'll need to remove this line that sets the context to CONTEXT_CHROME (context defaults to content), and then add this line back after you execute the GaiaLockScreen.unlock() method.

@@ +92,5 @@
>              log.info("%s | Application pid: %d",
>                       os.path.basename(__file__),
>                       self.runner.process_handler.pid)
> +            import time
> +            time.sleep(15)

If this is still necessary after fixing the context problem above, we'll have to add code to wait for the homescreen.  See http://mxr.mozilla.org/mozilla-central/source/testing/web-platform/harness/wptrunner/browsers/b2g.py#233 for an example of how this can be done.

Comment 16

[Armen [:armenzg]]

Attachment: mulet.diff

I was loosing hope and was going to study deeper!

It works (I see the screen being unlocked) and it seems the sleep is not needed.

I'm now hitting:
1410557694318	Marionette	INFO	sendToClient: {"from":"0","error":{"message":"NS_ERROR_FILE_NOT_FOUND: Component returned failure code: 0x80520012 (NS_ERROR_FILE_NOT_FOUND) [nsIXPCComponents_Utils.import]","status":17,"stacktrace":"execute_script @b2g_desktop.py, line 57\ninline javascript, line 100\nsrc: \"Cu.import(\"chrome://reftest/content/reftest.jsm\", reftest);\""}}, {ebb7e00c-beeb-449b-80a3-059b38009f8a}, {ebb7e00c-beeb-449b-80a3-059b38009f8a}
PROCESS-CRASH | b2g_desktop.py | application crashed [unknown top frame]
Crash dump filename: /tmp/tmpNLLpCD/minidumps/05be21c9-af6e-d77c-2a4fd4ca-79da497b.dmp
No symbols path given, can't process dump.
MINIDUMP_STACKWALK not set, can't process dump.
Traceback (most recent call last):
  File "reftest/runreftestb2g.py", line 611, in <module>
    sys.exit(main())
  File "reftest/runreftestb2g.py", line 606, in main
    return run_desktop_reftests(parser, options, args)
  File "/home/armenzg/moz/tmp/mulet/reftest/b2g_desktop.py", line 196, in run_desktop_reftests
    sys.exit(reftest.run_tests(args[0], options))
  File "/home/armenzg/moz/tmp/mulet/reftest/b2g_desktop.py", line 104, in run_tests
    self.run_marionette_script()
  File "/home/armenzg/moz/tmp/mulet/reftest/b2g_desktop.py", line 57, in run_marionette_script
    self.marionette.execute_script(self.test_script)
  File "/home/armenzg/moz/tmp/mulet/marionette/marionette/marionette.py", line 1205, in execute_script
    filename=os.path.basename(frame[0]))
  File "/home/armenzg/moz/tmp/mulet/marionette/marionette/decorators.py", line 35, in _
    return func(*args, **kwargs)
  File "/home/armenzg/moz/tmp/mulet/marionette/marionette/marionette.py", line 638, in _send_message
    self._handle_error(response)
  File "/home/armenzg/moz/tmp/mulet/marionette/marionette/marionette.py", line 686, in _handle_error
    raise errors.JavascriptException(message=message, status=status, stacktrace=stacktrace)
marionette.errors.JavascriptException: JavascriptException: NS_ERROR_FILE_NOT_FOUND: Component returned failure code: 0x80520012 (NS_ERROR_FILE_NOT_FOUND) [nsIXPCComponents_Utils.import]
stacktrace:
	execute_script @b2g_desktop.py, line 57
	inline javascript, line 100
	src: "Cu.import("chrome://reftest/content/reftest.jsm", reftest);"

Comment 17

[Armen [:armenzg]]

I have managed to run reftests. [1]

We now have to figure out where we should put the gaia lock screen JS code and meanwhile I will work locally on my build environment to generate reftest.jsm when building mulet.

I put the gaia lock screen JS code beside the b2g_desktop.py code. Let me know if that would be the right place.

[1]
* I downloaded https://pvtbuilds.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/mozilla-central-emulator/20140912061053/gaia-tests.zip
* I copied reftest.jsm from it
cp b2g_tests/reftest/reftest/chrome/reftest/content/reftest.jsm mulet/reftest/reftest/chrome/reftest/content/

Comment 18

[Jonathan Griffin (:jgriffin)]

> I put the gaia lock screen JS code beside the b2g_desktop.py code. Let me know if that would be the 
> right place.

Yes, I think that's fine.

Comment 19

[Armen [:armenzg]]

Attachment: mulet_reftests.diff

armenzg-thinkpad mozilla-central hg:[default!] $ find obj-dir-mulet/ -name reftest.js
obj-dir-mulet/dist/bin/chrome/reftest/content/reftest.js
armenzg-thinkpad mozilla-central hg:[default!] $ find obj-dir-mulet/ -name gaia_lock_screen.js
obj-dir-mulet/_tests/reftest/gaia_lock_screen.js

Pushed to try:
https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=ae607740c4f4

Comment 20

[Armen [:armenzg]]

Attachment: mulet_reftests.diff

This is the proper patch. I pushed something wrong:
https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=69fa463786c1

Comment 21

[Ed Morley [:emorley]]

Comment on attachment 8490106 [details] [diff] [review]
mulet_reftests.diff

Review of attachment 8490106 [details] [diff] [review]:
-----------------------------------------------------------------

::: layout/tools/reftest/b2g_desktop.py
@@ +61,5 @@
>          env = self.buildBrowserEnv(options, self.profile.profile)
> +
> +        # FIXME
> +        if "MOZ_DISABLE_NONLOCAL_CONNECTIONS" in env:
> +            del env["MOZ_DISABLE_NONLOCAL_CONNECTIONS"]

This will need to be removed for the jobs to be unhidden, seeing as this is a new job, per:
https://wiki.mozilla.org/Sheriffing/Job_Visibility_Policy#8.29_Must_avoid_patterns_known_to_cause_non_deterministic_failures

Comment 22

[Armen [:armenzg]]

The build seems to generate something that works.

You can reproduce it locally by following the steps below [1]

With regards to mozharness changes, for now it seems that I need to:
1) install marionette
2) use runreftestb2g.py.
3) append --desktop
4) append --profile
The Firefox's mozharness reftest config is not good as is.

On another note, are the mulet mochitest jobs calling the right thing?
They seem to be calling mochitest/runtests.py instead of mochitest/runtestsb2g.py [3]

[1]
Check out a local clone of gaia and make a gaia profile (by running 'make' in 
the gaia dir)
wget http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-69fa463786c1/try-linux64-mulet/firefox-35.0a1.en-US.linux-x86_64.tests.zip
wget http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-69fa463786c1/try-linux64-mulet/firefox-35.0a1.en-US.linux-x86_64.tar.bz2
wget http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-69fa463786c1/try-linux64-mulet/jsshell-linux-x86_64.zip
wget http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-69fa463786c1/try-linux64-mulet/firefox-35.0a1.en-US.linux-x86_64.crashreporter-symbols.zip
unzip -q firefox-35.0a1.en-US.linux-x86_64.tests.zip
tar xvfj firefox-35.0a1.en-US.linux-x86_64.tar.bz2
virtualenv temp_venv
source temp_venv/bin/activate
pip install -e marionette
python reftest/runreftestb2g.py --desktop --profile=/home/armenzg/repos/branches/gaia/profile --appname=firefox/firefox-bin reftest/tests/layout/reftests/reftest.list

[2]
python scripts/desktop_unittest.py --cfg unittests/linux_unittest.py --reftest-suite reftest --blob-upload-branch mozilla-central --download-symbols ondemand --test-url ~/moz/tmp/mulet_latest/firefox-35.0a1.en-US.linux-x86_64.tests.zip --installer-url ~/moz/tmp/mulet_latest/firefox-35.0a1.en-US.linux-x86_64.tar.bz2 --cfg developer_config.py --jsshell-url ~/moz/tmp/mulet_latest/jsshell-linux-x86_64.zip


python', '-u', '/home/armenzg/repos/mozharness/build/tests/reftest/runreftestb2g.py', '--appname=/home/armenzg/repos/mozharness/build/application/firefox/firefox', '--utility-path=tests/bin', '--extra-profile-file=tests/bin/plugins', '--symbols-path=/home/armenzg/moz/tmp/mulet_latest/firefox-35.0a1.en-US.linux-x86_64.crashreporter-symbols.zip', 'tests/reftest/tests/layout/reftests/reftest.list'

[3]
http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-69fa463786c1/try-linux64-mulet/try_ubuntu64_vm_mulet_test-mochitest-1-bm115-tests1-linux64-build149.txt.gz
10:05:45     INFO - Calling ['/builds/slave/test/build/venv/bin/python', '-u', '/builds/slave/test/build/tests/mochitest/runtests.py', '--total-chunks', '5', '--this-chunk', '1', '--appname=/builds/slave/test/build/application/firefox/firefox', '--utility-path=tests/bin', '--extra-profile-file=tests/bin/plugins', '--symbols-path=https://ftp-ssl.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-69fa463786c1/try-linux64-mulet/firefox-35.0a1.en-US.linux-x86_64.crashreporter-symbols.zip', '--certificate-path=tests/certs', '--autorun', '--close-when-done', '--console-level=INFO', '--setpref=webgl.force-enabled=true', '--quiet', '--log-raw=/builds/slave/test/build/blobber_upload_dir/raw_structured_logs.log', '--use-test-media-devices', '--chunk-by-dir=4'] with output_timeout 1000

Comment 23

[Jonathan Griffin (:jgriffin)]

> On another note, are the mulet mochitest jobs calling the right thing?
> They seem to be calling mochitest/runtests.py instead of mochitest/runtestsb2g.py [3]

Probably they're not.  Those jobs were setup with people unfamiliar with the way we run mochitests on B2G, so what we have running is the desktop flavor.  This is fine, except that those tests don't run OOP, which we probably want.  So, I think we'll need to move to B2G-style mochitests for Mulet.

Comment 24

[Armen [:armenzg]]

Jordan, I have managed to the most extent be able to use desktop_unittest.py to run mulet reftests, however, there's one step that does not fit properly inside of desktop_unittest.py.
The step requires to checkout gaia and run a "make" command.

What do you suggest doing?
I'm thinking of creating a mulet_unittest.py and have a simplified version of desktop_unittest.py that also checkout gaia.

I'm also considering a few days experiment where the script and config to deal with mulet lives in the tree and the mozharness code does a very setup and lets the script from the tree do the rest.
This would lay ground for a data point for next week's mozharness conversations.

What do you think?

Comment 25

[Jonathan Griffin (:jgriffin)]

Can we use b2g_desktop_unittest instead of desktop_unittest?

Comment 26

[Jonathan Griffin (:jgriffin)]

(In reply to Jonathan Griffin (:jgriffin) from comment #25)
> Can we use b2g_desktop_unittest instead of desktop_unittest?

The way these tests run is much more similar to b2gdesktop than it is to Firefox.

Comment 27

[Armen [:armenzg]]

Alexandre,
To get this running we need to checkout gaia and make a profile.

Which branch should we test against as we move through the release channels?
Does this look correct?

* m-i, m-c, fx-team, try et trunk trees:
** http://hg.mozilla.org/integration/gaia-central
* m-a
** http://hg.mozilla.org/integration/gaia-2_1
* Mozilla-B2g32-v2.0
** http://hg.mozilla.org/integration/gaia-2_0
and on...

I'm not saying that we will enable it yet on those branches but I want to make sure that we want to follow the right versions versus sticking to gaia-central across trees.

Comment 28

[Jonathan Griffin (:jgriffin)]

(In reply to Armen Zambrano - Automation & Tools Engineer (:armenzg) from comment #27)
> Alexandre,
> To get this running we need to checkout gaia and make a profile.
> 
> Which branch should we test against as we move through the release channels?
> Does this look correct?
> 
> * m-i, m-c, fx-team, try et trunk trees:
> ** http://hg.mozilla.org/integration/gaia-central
> * m-a
> ** http://hg.mozilla.org/integration/gaia-2_1
> * Mozilla-B2g32-v2.0
> ** http://hg.mozilla.org/integration/gaia-2_0
> and on...
> 
> I'm not saying that we will enable it yet on those branches but I want to
> make sure that we want to follow the right versions versus sticking to
> gaia-central across trees.

See e.g., http://hg.mozilla.org/build/mozharness/file/0f41dc44292e/scripts/gaia_unit.py.  We have a GaiaTest mixin that automatically clones the correct branch of gaia:  http://hg.mozilla.org/build/mozharness/file/0f41dc44292e/mozharness/mozilla/testing/gaia_test.py  It also has a 'make_gaia' method that builds a gaia profile.

Comment 29

[Armen [:armenzg]]

Attachment: wip - mulet mozharness

With this patch and command I can get to checkout gaia and make the profile.

python scripts/mulet_unittest.py --cfg b2g/desktop_automation_config.py --blob-upload-branch mozilla-central --download-symbols ondemand --test-url ~/moz/tmp/mulet_latest/firefox-35.0a1.en-US.linux-x86_64.tests.zip --installer-url ~/moz/tmp/mulet_latest/firefox-35.0a1.en-US.linux-x86_64.tar.bz2 --cfg b2g/mulet_config.py --cfg developer_config.py

More next week.

Comment 30

[Jordan Lund (:jlund)]

> The step requires to checkout gaia and run a "make" command.
> 
> What do you suggest doing?
> I'm thinking of creating a mulet_unittest.py and have a simplified version
> of desktop_unittest.py that also checkout gaia.

inheriting from GaiaTest should allow you to do that. however I'm not sure if you need all that it comes with. it looks like there is a GaiaMixin that may do exactly what you want: clone/make

marionette already avails of that directly: http://mxr.mozilla.org/build/source/mozharness/scripts/marionette.py#31

GaiaMixin: http://mxr.mozilla.org/build/source/mozharness/mozharness/mozilla/gaia.py#12

so you can do what marionette did and add it in to desktop_unittest.py. maybe b2g_desktop_unittest.py is more appropriate like jgriffin suggested?

> 
> I'm also considering a few days experiment where the script and config to
> deal with mulet lives in the tree and the mozharness code does a very setup
> and lets the script from the tree do the rest.
> This would lay ground for a data point for next week's mozharness
> conversations.

sure. depends on how we end up wanting to implement this but it could be a cool experiment :)

does this help?

Comment 31

[Alexandre Poirot [:ochameau]]

(In reply to Armen Zambrano - Automation & Tools Engineer (:armenzg) from comment #27)
> Alexandre,
> To get this running we need to checkout gaia and make a profile.
> 
> Which branch should we test against as we move through the release channels?

We should be doing what we are already doing for other b2g projects (b2g-desktop/device/emulator).
(It looks like you already found the code that does that...)

> Does this look correct?

Otherwise, yes at first sight it looks correct.
(Just note that m-a won't stick to 2.1 and will switch to 2.2 someday)

Comment 32

[Armen [:armenzg]]

I have tried various way on how to re-use the right class, however, there are a bunch of issues with the various Gaia related classes.

I'm going to try not do any re-factoring, however, it has to be dealt in the future (bug 1071219).

Comment 33

[Armen [:armenzg]]

Attachment: wip - mulet mozharness

This can be run like this:
wget http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-69fa463786c1/try-linux64-mulet/firefox-35.0a1.en-US.linux-x86_64.tests.zip
wget http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-69fa463786c1/try-linux64-mulet/firefox-35.0a1.en-US.linux-x86_64.tar.bz2
wget http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-69fa463786c1/try-linux64-mulet/firefox-35.0a1.en-US.linux-x86_64.crashreporter-symbols.zip
hg clone http://hg.mozilla.org/integration/gaia-central gaia
python scripts/mulet_unittest.py --cfg b2g/generic_config.py --cfg b2g/mulet_config.py --blob-upload-branch mozilla-central --download-symbols ondemand --test-url firefox-35.0a1.en-US.linux-x86_64.tests.zip --installer-url firefox-35.0a1.en-US.linux-x86_64.tar.bz2 --cfg developer_config.py --test-suite reftest --test-manifest tests/layout/reftests/reftest.list

Known issues:
* The gaia checkout is never clobbered
* It does not yet checkout gaia by itself

I'm requesting for feedback so at least I can get an idea that I'm not too off from your expectations.

NOTE: I have tried subclassing many other classes with much pain and little luck. I hope this can meet the expectations.

Comment 34

[Armen [:armenzg]]

Attachment: [checked-in] mulet_in_tree_configs.diff

With the previous patch we can "run_file_names" from the tree.

Comment 35

[Armen [:armenzg]]

Attachment: wip - mozharness_mulet.diff

This patch can now checkout gaia for me. It will need to test other scripts that make use of GaiaTest and GaiaMixin.

Instructions on how to run this locally [1]

Known issues:
* The gaia checkout and the gaia profile are never clobbered
* [FIXED] It does not yet checkout gaia by itself

[1]
wget http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-69fa463786c1/try-linux64-mulet/firefox-35.0a1.en-US.linux-x86_64.tests.zip
wget http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-69fa463786c1/try-linux64-mulet/firefox-35.0a1.en-US.linux-x86_64.tar.bz2
wget http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-69fa463786c1/try-linux64-mulet/firefox-35.0a1.en-US.linux-x86_64.crashreporter-symbols.zip
python scripts/mulet_unittest.py --cfg b2g/generic_config.py --cfg b2g/mulet_config.py --blob-upload-branch mozilla-central --download-symbols ondemand --test-url firefox-35.0a1.en-US.linux-x86_64.tests.zip --installer-url firefox-35.0a1.en-US.linux-x86_64.tar.bz2 --cfg developer_config.py --test-suite reftest --test-manifest tests/layout/reftests/reftest.list --gaia-repo http://hg.mozilla.org/integration/gaia-central

Comment 36

[Armen [:armenzg]]

Attachment: [checked-in] add mulet reftests on Ash

Comment 37

[Jordan Lund (:jlund)]

Comment on attachment 8494686 [details] [diff] [review]
[checked-in] add mulet reftests on Ash

Review of attachment 8494686 [details] [diff] [review]:
-----------------------------------------------------------------

lgtm. suppose we will do a tbpl patch after?

Comment 38

[Jordan Lund (:jlund)]

Comment on attachment 8494540 [details] [diff] [review]
wip - mozharness_mulet.diff

Review of attachment 8494540 [details] [diff] [review]:
-----------------------------------------------------------------

lrgtm :D :D
one high level question is how come you did not do to B2GDesktopTest what MarionetteTest did? I know you mentioned that you were having problems but could you not have just added GaiaMixin to B2GDesktopTest, overwritten make_gaia like you did in mulet_unittest.py, and extended the other mulet_unittest.py methods (run_test and query_abs_dirs) to support the mulet edge case.

On one argument, classes are getting overloaded with logic, on the other, we have too many mixins and subclasses. Not sure I know what we should do so just polling your opinion.

Looks great :)

::: mozharness/mozilla/gaia.py
@@ +17,5 @@
> +     }],
> +    [["--gaia-repo"],
> +    {"action": "store",
> +     "dest": "gaia_repo",
> +     "default": None,

I don't think the 'default' lines are needed. if not given, I think it will default to None

@@ +239,5 @@
> +                # gaia_branch by default is set to default
> +                self.fatal("You're trying to run this outside of buildbot, " \
> +                    "therefore, you need to specify --gaia-repo.")
> +
> +    def pull(self, **kwargs):

looks like marionette and GaiaTest have their own pull action. In which case the preflight_pull here will be called. I guess that's OK? Maybe even a good thing

@@ +250,5 @@
> +        dirs = self.query_abs_dirs()
> +        dest = dirs['abs_gaia_dir']
> +        repo = {}
> +
> +        if self.buildbot_config is not None:

same as 'if self.buildbot_config'.

I think we normally: 'self.buildbot_config and 'properties' in self.buildbot_config:'

@@ +261,5 @@
> +        else:
> +            repo = {
> +                'repo_path': self.config['gaia_repo'],
> +                'revision': 'default',
> +                'branch': self.config['gaia_branch']

should we add 'gaia_branch to preflight_pull like gaia_repo?

::: mozharness/mozilla/testing/gaia_test.py
@@ +76,5 @@
>           "help": "Number of this chunk",
>           }
>      ]] + copy.deepcopy(testing_config_options) + \
> +        copy.deepcopy(blobupload_config_options) + \
> +        copy.deepcopy(gaia_config_options)

good call :)

::: mozharness/mozilla/testing/testbase.py
@@ +113,5 @@
>                                              create_parent_dir=create_parent_dir,
>                                              error_level=error_level,
>                                              exit_code=exit_code)
>  
> +    def query_value(self, key):

I suspect that as you go, this method will make it closer and closer to the internals of mozharness. :)

::: scripts/b2g_desktop_unittest.py
@@ +70,5 @@
>          {'substr': 'Timeout waiting for marionette on port', 'level': ERROR},
>          {'regex': re.compile(r'''(Timeout|NoSuchAttribute|Javascript|NoSuchElement|XPathLookup|NoSuchWindow|StaleElement|ScriptTimeout|ElementNotVisible|NoSuchFrame|InvalidElementState|NoAlertPresent|InvalidCookieDomain|UnableToSetCookie|InvalidSelector|MoveTargetOutOfBounds)Exception'''), 'level': ERROR},
>      ]
>  
> +    def __init__(self, config=None, require_config_file=False):

I think we need to add GaiaMixin to this class no?

@@ +75,2 @@
>          super(B2GDesktopTest, self).__init__(
> +            config_options=self.config_options + copy.deepcopy(config),

if config is None, it will error trying to concat to a list

@@ +99,5 @@
>          self.test_url = c.get('test_url')
>          self.test_manifest = c.get('test_manifest')
>  
> +        suite = self.config['test_suite']
> +        if suite not in self.test_suites:

we want to support tree_config['suite'] or self.config['test_suite'] right? so if it's the former on a given script run, this test will always fatal right?

That is unless we pass suite to self.config *and* self.tree_config every time. but if we are moving to tree_config, I think we should drop self.config['suite'] altogether no?

::: scripts/mulet_unittest.py
@@ +18,5 @@
> +from mozharness.base.transfer import TransferMixin
> +from mozharness.mozilla.gaia import GaiaMixin, gaia_config_options
> +from b2g_desktop_unittest import B2GDesktopTest
> +
> +class MuletUnittest(B2GDesktopTest, GaiaMixin, TransferMixin):

since pull is defined in GaiaMixin, I think we should add GaiaMixin to its definition as I mentioned above

@@ +22,5 @@
> +class MuletUnittest(B2GDesktopTest, GaiaMixin, TransferMixin):
> +    test_suites = ('reftest',)
> +
> +    def __init__(self):
> +        super(MuletUnittest, self).__init__(config=gaia_config_options)

not sure if config is the best name for this. could we add options to the name?

@@ +52,5 @@
> +
> +    def preflight_run_tests(self):
> +        if self.config.get("binary_path"):
> +            self.binary_path = self.config["binary_path"]
> +        assert self.binary_path, \

asserts will skip log placement. I think self.fatal would be better as this is something we can only test at runtime

Comment 39

[Armen [:armenzg]]

Comment on attachment 8494686 [details] [diff] [review]
[checked-in] add mulet reftests on Ash

https://hg.mozilla.org/build/buildbot-configs/rev/93e40fcc9464

Comment 40

[Armen [:armenzg]]

Comment on attachment 8494540 [details] [diff] [review]
wip - mozharness_mulet.diff

Review of attachment 8494540 [details] [diff] [review]:
-----------------------------------------------------------------

> lrgtm :D :D
> one high level question is how come you did not do to B2GDesktopTest what
> MarionetteTest did? I know you mentioned that you were having problems but
> could you not have just added GaiaMixin to B2GDesktopTest, overwritten
> make_gaia like you did in mulet_unittest.py, and extended the other
> mulet_unittest.py methods (run_test and query_abs_dirs) to support the mulet
> edge case.
> 
I believe my original thought was not to mix mulet jobs with b2g jobs too much, however, as you can see we have already set a dependency on b2g desktop.

I would like to request for a review for my first approach so we can land it while I give a shot again at not needing mulet_unittest.py.

> On one argument, classes are getting overloaded with logic, on the other, we
> have too many mixins and subclasses. Not sure I know what we should do so
> just polling your opinion.

In my opinion, the long term solution is moving to writing scripts from the tree.
On the mean time, we should try to have code that makes it easy to write different scripts.

For instance, the marionette script had to redo a lot of functionality since it was not inside of GaiaMixin.
I think what would help here is if we had a clear separation of what goes into a Mixin and move it there.
For instance, GaiaTest should have defined whatever actions is overwriting and put everything else into GaiaMixin.
I did this a bit in my patch but not completely (e.g. query_abs_dir, extract_xre).

I'm also convinced that we should start thinking about using the concept of super reviewers to ensure that we have consistency and increase knowledge to make better architectural designs.
It is an area that we need due to the complexity of mozharness and one that not too many want to go to (more review load).
If we had mozharness modularized (as in proposal) we would have less need for actions, subclassing and mixins which would drop the need for super reviews.

::: mozharness/mozilla/gaia.py
@@ +261,5 @@
> +        else:
> +            repo = {
> +                'repo_path': self.config['gaia_repo'],
> +                'revision': 'default',
> +                'branch': self.config['gaia_branch']

gaia_branch has a default value and that is why I did not query, however, I would not mind adding it to be explicit (we never know if the default would change).

::: scripts/b2g_desktop_unittest.py
@@ +75,2 @@
>          super(B2GDesktopTest, self).__init__(
> +            config_options=self.config_options + copy.deepcopy(config),

I fixed this in a following patch.

@@ +99,5 @@
>          self.test_url = c.get('test_url')
>          self.test_manifest = c.get('test_manifest')
>  
> +        suite = self.config['test_suite']
> +        if suite not in self.test_suites:

In this case, all we're trying to do is make sure that --test-suite $suite is inside of the list of approved suites. In the case of MuletUnittests we currently only support reftests.

::: scripts/mulet_unittest.py
@@ +22,5 @@
> +class MuletUnittest(B2GDesktopTest, GaiaMixin, TransferMixin):
> +    test_suites = ('reftest',)
> +
> +    def __init__(self):
> +        super(MuletUnittest, self).__init__(config=gaia_config_options)

Sure.

@@ +52,5 @@
> +
> +    def preflight_run_tests(self):
> +        if self.config.get("binary_path"):
> +            self.binary_path = self.config["binary_path"]
> +        assert self.binary_path, \

Fixed in next patch.

Comment 41

[Armen [:armenzg]]

Attachment: mulet_reftests.diff

This patch works on Ash, including for b2g desktop and gaia unit jobs.

As I mentioned, I would like to land as-is and do a bit of work to refactor it properly.

Comment 42

[Armen [:armenzg]]

jgriffin, what do we need to do with regards to this comment?

(In reply to Ed Morley [:edmorley] from comment #21)
> Comment on attachment 8490106 [details] [diff] [review]
> mulet_reftests.diff
> 
> Review of attachment 8490106 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> ::: layout/tools/reftest/b2g_desktop.py
> @@ +61,5 @@
> >          env = self.buildBrowserEnv(options, self.profile.profile)
> > +
> > +        # FIXME
> > +        if "MOZ_DISABLE_NONLOCAL_CONNECTIONS" in env:
> > +            del env["MOZ_DISABLE_NONLOCAL_CONNECTIONS"]
> 
> This will need to be removed for the jobs to be unhidden, seeing as this is
> a new job, per:
> https://wiki.mozilla.org/Sheriffing/Job_Visibility_Policy#8.
> 29_Must_avoid_patterns_known_to_cause_non_deterministic_failures

Comment 43

[Jonathan Griffin (:jgriffin)]

(In reply to Armen Zambrano - Automation & Tools Engineer (:armenzg) from comment #42)
> jgriffin, what do we need to do with regards to this comment?
> 
> (In reply to Ed Morley [:edmorley] from comment #21)
> > Comment on attachment 8490106 [details] [diff] [review]
> > mulet_reftests.diff
> > 
> > Review of attachment 8490106 [details] [diff] [review]:
> > -----------------------------------------------------------------
> > 
> > ::: layout/tools/reftest/b2g_desktop.py
> > @@ +61,5 @@
> > >          env = self.buildBrowserEnv(options, self.profile.profile)
> > > +
> > > +        # FIXME
> > > +        if "MOZ_DISABLE_NONLOCAL_CONNECTIONS" in env:
> > > +            del env["MOZ_DISABLE_NONLOCAL_CONNECTIONS"]
> > 
> > This will need to be removed for the jobs to be unhidden, seeing as this is
> > a new job, per:
> > https://wiki.mozilla.org/Sheriffing/Job_Visibility_Policy#8.
> > 29_Must_avoid_patterns_known_to_cause_non_deterministic_failures

Good question.  The issue is that Gaia makes some remote http requests on startup that we have no way to disable.  AFAICT, these don't impact the tests, but who knows for sure.  I'll pass this back to Ed...Ed, if you feel strongly about this, we'll need to file bugs for devs to implement ways to skip these.

Comment 44

[Armen [:armenzg]]

Attachment: [checked-in] mozharness_mulet.diff

I attached an old patch.

Comment 45

[Ed Morley [:emorley]]

(In reply to Jonathan Griffin (:jgriffin) from comment #43)
> > > > +        if "MOZ_DISABLE_NONLOCAL_CONNECTIONS" in env:
> > > > +            del env["MOZ_DISABLE_NONLOCAL_CONNECTIONS"]
> > > 
> > > This will need to be removed for the jobs to be unhidden, seeing as this is
> > > a new job, per:
> > > https://wiki.mozilla.org/Sheriffing/Job_Visibility_Policy#8.
> > > 29_Must_avoid_patterns_known_to_cause_non_deterministic_failures
> 
> Good question.  The issue is that Gaia makes some remote http requests on
> startup that we have no way to disable.  AFAICT, these don't impact the
> tests, but who knows for sure.  I'll pass this back to Ed...Ed, if you feel
> strongly about this, we'll need to file bugs for devs to implement ways to
> skip these.

Remote requests on startup can still affect the reliability of the job - and worse, by having this disabled we won't detect future regressions for requests not at startup. Given that opt B2G mochitest jobs are successfully running without this disabled (http://mxr.mozilla.org/mozilla-central/source/testing/mochitest/runtestsb2g.py#163) I would like to see at least the opt builds running without undefining MOZ_DISABLE_NONLOCAL_CONNECTIONS, if that's ok?

Comment 46

[Jonathan Griffin (:jgriffin)]

Understood.  In that case, can you remove those bits Armen, and file a bug for whichever request it is that causes the test to crash?  I can help direct the bug to the right component if I can see a test log for it.

Comment 47

[Jordan Lund (:jlund)]

Comment on attachment 8495506 [details] [diff] [review]
[checked-in] mozharness_mulet.diff

Review of attachment 8495506 [details] [diff] [review]:
-----------------------------------------------------------------

lgtm :) if it works on ash, shweet.

But I am not sure if it is all working. My main concern is that this still uses a 'config' arg in init of scripts/b2g_desktop_unittest.py but scripts/mulet_unittest.py uses 'options' now.

::: configs/b2g/generic_config.py
@@ +1,1 @@
> +# This is a config with generic releng related values 

remind me, what's the plan here? are you going to rip out duplicate items from b2g configs and call this after a buildbot-config patch?

I wonder if we should be creating this file in tree and adding it to self.tree_config rather than within mozharn repo. Apologies if you have explained this already.

::: mozharness/mozilla/testing/gaia_test.py
@@ -154,5 @@
> -
> -        self.clone_gaia(dest, repo,
> -                        use_gaia_json=self.buildbot_config is not None)
> -
> -        super(GaiaTest, self).pull(**kwargs)

what pull was this super calling? looks like we are no longer doing that after your patch.

::: scripts/b2g_desktop_unittest.py
@@ +70,5 @@
>          {'substr': 'Timeout waiting for marionette on port', 'level': ERROR},
>          {'regex': re.compile(r'''(Timeout|NoSuchAttribute|Javascript|NoSuchElement|XPathLookup|NoSuchWindow|StaleElement|ScriptTimeout|ElementNotVisible|NoSuchFrame|InvalidElementState|NoAlertPresent|InvalidCookieDomain|UnableToSetCookie|InvalidSelector|MoveTargetOutOfBounds)Exception'''), 'level': ERROR},
>      ]
>  
> +    def __init__(self, config=[], require_config_file=False):

I think because of implentation of python, we have to assign config with [] outside of of the method signature or spurious things can happen: http://stackoverflow.com/questions/9887180/the-value-of-an-empty-list-in-function-parameter-example-here

also, I'm guessing this is supposed to be 'options' now after your change in mulet_unittest.py

::: scripts/mulet_unittest.py
@@ +55,5 @@
> +        # of download-and-extract()
> +        if self.config.get("binary_path"):
> +            self.binary_path = self.config["binary_path"]
> +
> +        if not self.binary_path:

should we do this instead incase we have self.binary_path already set?

if not self.binary_path:
    if self.config.get("binary_path"):
        self.binary_path = self.config["binary_path"]
    else:
        self.fatal("Use --binary-path as it is needed for _query_abs_dir().")

it might not matter. the above is the idiom we normally do with self attrs and self.config. doesn't block

Comment 48

[Armen [:armenzg]]

Comment on attachment 8495506 [details] [diff] [review]
[checked-in] mozharness_mulet.diff

Review of attachment 8495506 [details] [diff] [review]:
-----------------------------------------------------------------

> But I am not sure if it is all working. My main concern is that this still
> uses a 'config' arg in init of scripts/b2g_desktop_unittest.py but
> scripts/mulet_unittest.py uses 'options' now.
> 
You're right. This was fixed on a posterior push to Ash. I have kept on working on a proper refactoring for this.

::: configs/b2g/generic_config.py
@@ +1,1 @@
> +# This is a config with generic releng related values 

The plain is experiment with the idea that *all* if not most releng values could be contained on a generic config. Eventually, once we move things into the tree, I would like to make all scripts to point to such universal generic config and then specialize for the specific platform. This is just one step towards it.
I don't see this config in the tree since a lot of these values might be needed before the download-and-extract step happens.
I don't currently see the value of having this in the tree as of now, however, I'm happy to discuss it further now or later down the road.

::: mozharness/mozilla/testing/gaia_test.py
@@ -154,5 @@
> -
> -        self.clone_gaia(dest, repo,
> -                        use_gaia_json=self.buildbot_config is not None)
> -
> -        super(GaiaTest, self).pull(**kwargs)

For some reason I recall seeing the tools repo being checked out, however, I can't see "repos" being specified for marionete, b2g desktop or gaia unit.
I will review it again if I do need this.

::: scripts/b2g_desktop_unittest.py
@@ +70,5 @@
>          {'substr': 'Timeout waiting for marionette on port', 'level': ERROR},
>          {'regex': re.compile(r'''(Timeout|NoSuchAttribute|Javascript|NoSuchElement|XPathLookup|NoSuchWindow|StaleElement|ScriptTimeout|ElementNotVisible|NoSuchFrame|InvalidElementState|NoAlertPresent|InvalidCookieDomain|UnableToSetCookie|InvalidSelector|MoveTargetOutOfBounds)Exception'''), 'level': ERROR},
>      ]
>  
> +    def __init__(self, config=[], require_config_file=False):

That is insane!! That is a rather absurd behaviour of python.

I was thinking of fixing, however, after some thought, we never call an __init__ method of a script more than once.
Do you think I should still fix it?

::: scripts/mulet_unittest.py
@@ +55,5 @@
> +        # of download-and-extract()
> +        if self.config.get("binary_path"):
> +            self.binary_path = self.config["binary_path"]
> +
> +        if not self.binary_path:

What do you think of this instead? A bit cleaner; I think.

    def preflight_run_tests(self):
        # This is if we don't run all actions since we set this inside
        # of download-and-extract()
        if not self.binary_path:
            if self.config.get("binary_path"):
                self.binary_path = self.config["binary_path"]
            else:
                self.fatal("Use --binary-path as it is needed for _query_abs_dir().")

Comment 49

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/4282122a081e

Comment 50

[Armen [:armenzg]]

Attachment: [checked-in] bconfigs_enable_mulet_on_try.diff

Comment 51

[Armen [:armenzg]]

Comment on attachment 8495506 [details] [diff] [review]
[checked-in] mozharness_mulet.diff

https://hg.mozilla.org/build/mozharness/rev/b59a1a06dd1e

Comment 52

[Armen [:armenzg]]

You can now run mulet through mozharness.
You can try it out with:
python scripts/mulet_unittest.py --cfg b2g/generic_config.py --cfg b2g/mulet_config.py --test-suite reftest --test-manifest tests/layout/reftests/reftest.list --blob-upload-branch ash --download-symbols ondemand --installer-url http://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/ash-linux64-mulet/1412005020/firefox-35.0a1.en-US.linux-x86_64.tar.bz2 --test-url http://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/ash-linux64-mulet/1412005020/firefox-35.0a1.en-US.linux-x86_64.tests.zip --cfg developer_config.py --gaia-repo ~/repos/branches/gaia-central

(In reply to Jonathan Griffin (:jgriffin) from comment #46)
> Understood.  In that case, can you remove those bits Armen, and file a bug
> for whichever request it is that causes the test to crash?  I can help
> direct the bug to the right component if I can see a test log for it.

I have seen this happen locally [1]

While on Ash I see this instead [2]
https://tbpl.mozilla.org/php/getParsedLog.php?id=49121431&tree=Ash&full=1#error0

[1]
http://hg.mozilla.org/mozilla-central/file/6eb7121059fb/netwerk/base/src/nsSocketTransport2.cpp#l1184

16:38:06     INFO -  FATAL ERROR: Non-local network connections are disabled and a connection attempt to inapp-pay-test.paas.allizom.org (63.245.215.73) was made.
16:38:06     INFO -  You should only access hostnames available via the test networking proxy (if running mochitests) or from a test-specific httpd.js server (if running xpcshell tests). Browser services should be disabled or redirected to a local server.
16:38:06     INFO -  ###!!! [Child][MessageChannel] Error: Channel closing: too late to send/recv, messages will be lost
16:38:06     INFO -  [Child 8101] WARNING: pipe error (3): Connection reset by peer: file /builds/slave/ash-l64-mulet-0000000000000000/build/ipc/chromium/src/chrome/common/ipc_channel_posix.cc, line 451
16:38:06     INFO -  ###!!! [Child][MessageChannel] Error: Channel error: cannot send/recv
16:38:06  WARNING -  PROCESS-CRASH | b2g_desktop.pyc | application crashed [unknown top frame]
16:38:06     INFO -  Crash dump filename: /tmp/tmpc07zn1/minidumps/3b46a9ed-d7b0-b362-782e876c-53ad2a11.dmp
16:38:06     INFO -  No symbols path given, can't process dump.
16:38:06     INFO -  Traceback (most recent call last):
16:38:06     INFO -    File "runreftestb2g.py", line 611, in <module>
16:38:06     INFO -      sys.exit(main())
16:38:06     INFO -    File "runreftestb2g.py", line 606, in main
16:38:06     INFO -      return run_desktop_reftests(parser, options, args)
16:38:06     INFO -    File "/home/armenzg/repos/mozharness/build/tests/reftest/b2g_desktop.py", line 187, in run_desktop_reftests
16:38:06     INFO -      sys.exit(reftest.run_tests(args[0], options))
16:38:06     INFO -    File "/home/armenzg/repos/mozharness/build/tests/reftest/b2g_desktop.py", line 95, in run_tests
16:38:06     INFO -      self.run_marionette_script()
16:38:06     INFO -    File "/home/armenzg/repos/mozharness/build/tests/reftest/b2g_desktop.py", line 44, in run_marionette_script
16:38:06     INFO -      self.marionette.start_session()
16:38:06     INFO -    File "/home/armenzg/repos/mozharness/build/venv/local/lib/python2.7/site-packages/marionette/marionette.py", line 818, in start_session
16:38:06     INFO -      self.session = self._send_message('newSession', 'value')
16:38:06     INFO -    File "/home/armenzg/repos/mozharness/build/venv/local/lib/python2.7/site-packages/marionette/decorators.py", line 35, in _
16:38:06     INFO -      return func(*args, **kwargs)
16:38:06     INFO -    File "/home/armenzg/repos/mozharness/build/venv/local/lib/python2.7/site-packages/marionette/marionette.py", line 615, in _send_message
16:38:06     INFO -      response = self.client.send(message)
16:38:06     INFO -    File "/home/armenzg/repos/mozharness/build/venv/local/lib/python2.7/site-packages/marionette_transport/transport.py", line 85, in send
16:38:06     INFO -      self.connect()
16:38:06     INFO -    File "/home/armenzg/repos/mozharness/build/venv/local/lib/python2.7/site-packages/marionette_transport/transport.py", line 67, in connect
16:38:06     INFO -      self.sock.connect((self.addr, self.port))
16:38:06     INFO -    File "/usr/lib/python2.7/socket.py", line 224, in meth
16:38:06     INFO -      return getattr(self._sock,name)(*args)
16:38:06     INFO -  socket.error: [Errno 111] Connection refused

[2]
https://tbpl.mozilla.org/php/getParsedLog.php?id=49121431&tree=Ash&full=1#error0
09:44:19     INFO -  Traceback (most recent call last):
09:44:19     INFO -    File "runreftestb2g.py", line 611, in <module>
09:44:19     INFO -      sys.exit(main())
09:44:19     INFO -    File "runreftestb2g.py", line 606, in main
09:44:19     INFO -      return run_desktop_reftests(parser, options, args)
09:44:19     INFO -    File "/builds/slave/test/build/tests/reftest/b2g_desktop.py", line 187, in run_desktop_reftests
09:44:19     INFO -      sys.exit(reftest.run_tests(args[0], options))
09:44:19     INFO -    File "/builds/slave/test/build/tests/reftest/b2g_desktop.py", line 95, in run_tests
09:44:19     INFO -      self.run_marionette_script()
09:44:19     INFO -    File "/builds/slave/test/build/tests/reftest/b2g_desktop.py", line 45, in run_marionette_script
09:44:19     INFO -      self._unlockScreen()
09:44:19     INFO -    File "/builds/slave/test/build/tests/reftest/b2g_desktop.py", line 39, in _unlockScreen
09:44:19     INFO -      self.marionette.execute_async_script('GaiaLockScreen.unlock()')
09:44:19     INFO -    File "/builds/slave/test/build/venv/local/lib/python2.7/site-packages/marionette/marionette.py", line 1269, in execute_async_script
09:44:19     INFO -      filename=os.path.basename(frame[0]))
09:44:19     INFO -    File "/builds/slave/test/build/venv/local/lib/python2.7/site-packages/marionette/decorators.py", line 35, in _
09:44:19     INFO -      return func(*args, **kwargs)
09:44:19     INFO -    File "/builds/slave/test/build/venv/local/lib/python2.7/site-packages/marionette/marionette.py", line 638, in _send_message
09:44:19     INFO -      self._handle_error(response)
09:44:19     INFO -    File "/builds/slave/test/build/venv/local/lib/python2.7/site-packages/marionette/marionette.py", line 686, in _handle_error
09:44:19    ERROR -      raise errors.JavascriptException(message=message, status=status, stacktrace=stacktrace)
09:44:19    ERROR -  marionette.errors.JavascriptException: JavascriptException: TypeError: this.states.instance is null
09:44:19     INFO -  stacktrace:
09:44:19     INFO -  	@app://system.gaiamobile.org/js/lockscreen_window_manager.js, line 245
09:44:19    ERROR - Return code: 1

Comment 53

[Armen [:armenzg]]

The code used for the previous comment is this:
https://hg.mozilla.org/projects/ash/rev/6a0c55df4061

Which is the same as attachment 8490106 [details] [diff] [review] but by removing this:
::: layout/tools/reftest/b2g_desktop.py
@@ +61,5 @@
>          env = self.buildBrowserEnv(options, self.profile.profile)
> +
> +        # FIXME
> +        if "MOZ_DISABLE_NONLOCAL_CONNECTIONS" in env:
> +            del env["MOZ_DISABLE_NONLOCAL_CONNECTIONS"]

Comment 54

[Armen [:armenzg]]

jlund, I can't seem to find a way to mark a test job to not be run by default on try.
I can see "try_by_default" for build jobs but not for build jobs.

I could also hide mulet reftests on try until they're fixed.

Comment 55

[Jonathan Griffin (:jgriffin)]

I'll file a bug for #1.  #2 looks like a timing issue...the slave is probably slower than your machine, and possibly the lock screen hasn't appeared by the time we are trying to unlock it.  Does the problem on ash disappear if you sleep(15) before unlocking the lockscreen?  If so, we probably need to add some code to wait for the lock screen to appear before we attempt to unlock it.

Comment 56

[Armen [:armenzg]]

Attachment: mulet_reftests.diff

With this patch we can now hit my local issue:
https://tbpl.mozilla.org/php/getParsedLog.php?id=49211733&tree=Ash&full=1#error0

I guess we just have to wait now for bug 1053163.

Meanwhile let's get the try support so developers can fix it on try (attachment 8497039 [details] [diff] [review]).

Comment 57

[Jordan Lund (:jlund)]

Comment on attachment 8497039 [details] [diff] [review]
[checked-in] bconfigs_enable_mulet_on_try.diff

Review of attachment 8497039 [details] [diff] [review]:
-----------------------------------------------------------------

lgtm but can we make this a trychooser option? looking at this patch, I assume this will run by default if we -all on try. Which might upset sheriffs as, correct me if I am wrond, we shouldn't have any tests that are perma fail running by default on try.

Comment 58

[Armen [:armenzg]]

I'm aiming at hiding it since I don't know of a way to not trigger it by default.

Comment 59

[Chris Cooper [:coop] (he/him)]

Merged to production, and deployed.

Comment 60

[Jordan Lund (:jlund)]

(In reply to Armen Zambrano - Automation & Tools Engineer (:armenzg) from comment #58)
> I'm aiming at hiding it since I don't know of a way to not trigger it by
> default.

if you can not find a way to trigger it only via trychooser, I can only see an example for builds[1], hiding wfm. Sorry to stall you on this. Just trying to avoid adding unneeded jobs

[1] https://bug691177.bugzilla.mozilla.org/attachment.cgi?id=695029

Comment 61

[Jordan Lund (:jlund)]

(In reply to Armen Zambrano - Automation & Tools Engineer (:armenzg) from comment #54)
> jlund, I can't seem to find a way to mark a test job to not be run by
> default on try.
> I can see "try_by_default" for build jobs but not for build jobs.
> 
> I could also hide mulet reftests on try until they're fixed.

well that's embarrassing. just seeing this comment now. looks like I've been one step behind all along.

Comment 62

[Armen [:armenzg]]

Comment on attachment 8497039 [details] [diff] [review]
[checked-in] bconfigs_enable_mulet_on_try.diff

I did not find a way to disable a specific builder.
I only found a way to disable a whole platform by adding "try_by_default": False (e.g. u'linux64-mulet': ['Ubuntu VM 12.04 x64 Mulet try-nondefault'],).

I'm landing as-is and I hope to hide it on time once it shows up on tbpl.
https://hg.mozilla.org/build/buildbot-configs/rev/3bfd982c2211

Comment 63

[Chris Cooper [:coop] (he/him)]

Merged to production, and deployed.

Comment 64

[Armen [:armenzg]]

Comment on attachment 8497650 [details] [diff] [review]
mulet_reftests.diff

I think we should land this while it does not get bitrotten.

Comment 65

[Armen [:armenzg]]

Attachment: [checked-in] [buildbot-configs] enable mulet reftests on Cedar

This will allow developers to know the current state of mulet reftests without first having to push anything to try.

Comment 66

[Jonathan Griffin (:jgriffin)]

Comment on attachment 8497650 [details] [diff] [review]
mulet_reftests.diff

Review of attachment 8497650 [details] [diff] [review]:
-----------------------------------------------------------------

::: layout/tools/reftest/b2g_desktop.py
@@ +117,5 @@
>          # Turn off the locale picker screen
>          prefs["browser.firstrun.show.localepicker"] = False
> +        # FIXME: these cause Gaia not to launch
> +        #prefs["b2g.system_startup_url"] = "app://test-container.gaiamobile.org/index.html"
> +        #prefs["b2g.system_manifest_url"] = "app://test-container.gaiamobile.org/manifest.webapp"

Have we done a try run to make sure this doesn't break reftest on b2gdesktop?  I'm afraid the above lines might do that.

Comment 67

[Armen [:armenzg]]

(In reply to Jonathan Griffin (:jgriffin) from comment #66)
> Comment on attachment 8497650 [details] [diff] [review]
> mulet_reftests.diff
> 
> Review of attachment 8497650 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> ::: layout/tools/reftest/b2g_desktop.py
> @@ +117,5 @@
> >          # Turn off the locale picker screen
> >          prefs["browser.firstrun.show.localepicker"] = False
> > +        # FIXME: these cause Gaia not to launch
> > +        #prefs["b2g.system_startup_url"] = "app://test-container.gaiamobile.org/index.html"
> > +        #prefs["b2g.system_manifest_url"] = "app://test-container.gaiamobile.org/manifest.webapp"
> 
> Have we done a try run to make sure this doesn't break reftest on
> b2gdesktop?  I'm afraid the above lines might do that.

My bad. We will know tomorrow:
https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=3e3714577db2

Comment 68

[Armen [:armenzg]]

It does affect the b2g desktop jobs.
https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=3e3714577db2
https://treeherder.mozilla.org/ui/logviewer.html#?job_id=2253625&repo=try

 15:52:03 INFO - FATAL ERROR: Non-local network connections are disabled and a connection attempt to inapp-pay-test.paas.allizmozcrash INFO | Downloading symbols from: https://ftp-ssl.mozilla.org/pub/mozilla.org/b2g/try-builds/[e-mail]-3e3714577db2/try-linux64_gecko/b2g-35.0a1.en-US.linux-x86_64.crashreporter-symbols.zip 

jgriffin, what do you suggest I try next? Or leave this until bug 1053163 is fixed?

I've considered adding a --mulet parameter to the harness and use it for mulet jobs, hence not affecting the b2g desktop jobs.

Comment 69

[Jonathan Griffin (:jgriffin)]

Since this is just on cedar for now (or try) I think we should allow network connections for the purpose of greening up the tests themselves.   After bug 1053163 is addressed, we can look at disabling network connections again.

Comment 70

[Jordan Lund (:jlund)]

Comment on attachment 8500612 [details] [diff] [review]
[checked-in] [buildbot-configs] enable mulet reftests on Cedar

Review of attachment 8500612 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla-tests/b2g_config.py
@@ +2242,5 @@
>  exclude_suites(('linux32_gecko', 'ubuntu32_vm-b2gdt'), 'cedar', ('gaia-ui-test',), True, True)
>  exclude_suites(('macosx64_gecko', 'mountainlion-b2gdt'), 'cedar', ('gaia-ui-test',), True, True)
>  exclude_suites(('emulator', 'ubuntu64_vm-b2g-lg-emulator'), 'cedar', ('gaia-ui-test',), True, True)
>  
>  # Enable mulet reftests on Ash and Try

# and cedar

Comment 71

[Pete Moore [:pmoore][:pete]]

Committed code from this bug has been rolled out to production.

Comment 72

[Armen [:armenzg]]

Attachment: [m-c] patch to run mulet reftest: 1) allow external connections, 2) do not comment out gaia prefs and 3) add sleep

I want to see if I can create a patch that will not turn b2g desktop reftest jobs red for Cedar (like attachment 8497650 [details] [diff] [review] does):
https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=dd38ffbef696

I added back allowing external connections.

Comment 73

[Armen [:armenzg]]

Attachment: [m-c] patch to run mulet reftest: 1) add --mulet flag 2) allow external connections, 3) do not add gaia prefs and 4) add sleep

I'm back to this. I'm adding a --mulet option to runreftestb2g.py/b2g_desktop.py to allow us to have the necessary code to run mulet reftests without affecting the b2g desktop jobs.

Pushing this change to try:
https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=24fee2a32d3c

Also pushed without permitting external connections to see where we are:
https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=109394c6b860

STR:
python scripts/mulet_unittest.py --cfg b2g/generic_config.py \
--cfg b2g/mulet_config.py --test-suite reftest --test-manifest tests/layout/reftests/reftest.list \
--blob-upload-branch ash --download-symbols ondemand \
--installer-url TBD \
--test-url TBD \
--cfg developer_config.py --gaia-repo /path/to/gaia/centra/checkout

NOTE: Check-out gaia-central if you don't want the script to be checking out every time.

Comment 74

[Armen [:armenzg]]

Attachment: [m-c] patch to run mulet reftest: 1) add --mulet flag 2) allow external connections, 3) do not add gaia prefs and 4) add sleep

Updated patch.
Latest run on try looks good:
https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=07296ac81420 (pending desktop jobs)

There's some clean up of imports that are not used.

Comment 75

[Ed Morley [:emorley]]

Comment on attachment 8512212 [details] [diff] [review]
[m-c] patch to run mulet reftest: 1) add --mulet flag 2) allow external connections, 3) do not add gaia prefs and 4) add sleep

Review of attachment 8512212 [details] [diff] [review]:
-----------------------------------------------------------------

::: layout/tools/reftest/b2g_desktop.py
@@ +67,5 @@
> +
> +        if self.build_type == "mulet":
> +            # FIXME
> +            if "MOZ_DISABLE_NONLOCAL_CONNECTIONS" in env:
> +                del env["MOZ_DISABLE_NONLOCAL_CONNECTIONS"]

Is this still needed now that bug 1053163's PR landed?
If it is needed, could we make it mulet+debug? (Presuming it's green on opt like other B2G)

Comment 76

[Armen [:armenzg]]

(In reply to Ed Morley [:edmorley] from comment #75)
> Comment on attachment 8512212 [details] [diff] [review]
> [m-c] patch to run mulet reftest: 1) add --mulet flag 2) allow external
> connections, 3) do not add gaia prefs and 4) add sleep
> 
> Review of attachment 8512212 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> ::: layout/tools/reftest/b2g_desktop.py
> @@ +67,5 @@
> > +
> > +        if self.build_type == "mulet":
> > +            # FIXME
> > +            if "MOZ_DISABLE_NONLOCAL_CONNECTIONS" in env:
> > +                del env["MOZ_DISABLE_NONLOCAL_CONNECTIONS"]
> 
> Is this still needed now that bug 1053163's PR landed?
> If it is needed, could we make it mulet+debug? (Presuming it's green on opt
> like other B2G)

I have pushed separately to determine that:
https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=5fe267a972e5

Comment 77

[Armen [:armenzg]]

All jobs are green except for the mulet reftest ones:
* Log: http://ftp.mozilla.org/pub/mozilla.org/b2g/try-builds/[e-mail]-5fe267a972e5/try-linux64-mulet/try_ubuntu64_vm-mulet_test-reftest-bm117-tests1-linux64-build40.txt.gz
* Changes: https://hg.mozilla.org/try/rev/07296ac81420

I will run it locally and see what is going on.

16:59:19     INFO -  [3086] WARNING: Cannot parse integer value '800x1000' for --screen: 'glib warning', file /builds/slave/try-l64-mulet-0000000000000000/build/toolkit/xre/nsSigHandlers.cpp, line 129
16:59:19     INFO -  (firefox:3086): Gtk-WARNING **: Cannot parse integer value '800x1000' for --screen
16:59:19     INFO -  Xlib:  extension "RANDR" missing on display ":0".
16:59:19     INFO -  1414454351593	addons.xpi	WARN	Error loading bootstrap.js for [e-mail]: Error opening input stream (invalid filename?)
16:59:19     INFO -  1414454351594	addons.xpi	WARN	Add-on [e-mail] is missing bootstrap method install
16:59:19     INFO -  1414454351624	addons.xpi	WARN	Disabling foreign installed add-on [e-mail] in app-system-share
16:59:19     INFO -  1414454351631	addons.xpi	WARN	Add-on [e-mail] is missing bootstrap method startup
...
16:59:19     INFO -  1414454359735	Marionette	INFO	sendToClient: {"from":"0","error":{"message":"TypeError: doc is undefined","status":17,"stacktrace":"execute_script @b2g_desktop.py, line 57\ninline javascript, line 280\nsrc: \"undefined\""}}, {ae7ae623-0f8f-4b41-85a5-df8df50a4da7}, {ae7ae623-0f8f-4b41-85a5-df8df50a4da7}
16:59:19     INFO -  Traceback (most recent call last):
16:59:19     INFO -    File "runreftestb2g.py", line 633, in <module>
16:59:19     INFO -      sys.exit(main())
16:59:19     INFO -    File "runreftestb2g.py", line 628, in main
16:59:19     INFO -      return run_desktop_reftests(parser, options, args)
16:59:19     INFO -    File "/builds/slave/test/build/tests/reftest/b2g_desktop.py", line 208, in run_desktop_reftests
16:59:19     INFO -      sys.exit(reftest.run_tests(args[0], options))
16:59:19     INFO -    File "/builds/slave/test/build/tests/reftest/b2g_desktop.py", line 105, in run_tests
16:59:19     INFO -      self.run_marionette_script()
16:59:19     INFO -    File "/builds/slave/test/build/tests/reftest/b2g_desktop.py", line 57, in run_marionette_script
16:59:19     INFO -      self.marionette.execute_script(self.test_script)
16:59:19     INFO -    File "/builds/slave/test/build/venv/local/lib/python2.7/site-packages/marionette/marionette.py", line 1221, in execute_script
16:59:19     INFO -      filename=os.path.basename(frame[0]))
16:59:19     INFO -    File "/builds/slave/test/build/venv/local/lib/python2.7/site-packages/marionette/decorators.py", line 36, in _
16:59:19     INFO -      return func(*args, **kwargs)
16:59:19     INFO -    File "/builds/slave/test/build/venv/local/lib/python2.7/site-packages/marionette/marionette.py", line 638, in _send_message
16:59:19     INFO -      self._handle_error(response)
16:59:19     INFO -    File "/builds/slave/test/build/venv/local/lib/python2.7/site-packages/marionette/marionette.py", line 686, in _handle_error
16:59:19    ERROR -      raise errors.JavascriptException(message=message, status=status, stacktrace=stacktrace)
16:59:19    ERROR -  marionette.errors.JavascriptException: JavascriptException: TypeError: doc is undefined
16:59:19     INFO -  stacktrace:
16:59:19     INFO -  	execute_script @b2g_desktop.py, line 57
16:59:19     INFO -  	inline javascript, line 280
16:59:19     INFO -  	src: "undefined"

Comment 78

[Armen [:armenzg]]

I spotted the issue. I will put up a new patch.

Comment 79

[Armen [:armenzg]]

Attachment: [m-c] patch to run mulet reftest: 1) add --mulet flag 2) allow external connections, 3) do not add gaia prefs and 4) add sleep

I had forgotten to adjust the in-tree mozharness config to switch from --desktop to --mulet.

The new patch being tested:
https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=caf13ef7ff7c

Comment 80

[Armen [:armenzg]]

Attachment: m-c] patch to run mulet reftest: 1) add --mulet flag 2) allow external connections, 3) do not add gaia prefs and 4) add sleep

I attached the wrong patch. This is the right one.

We now can run lots of reftests, however, we time out after 2 hours.

I think we have these options:
- We probably have to break the reftests into various chunks
- Determine why we run so slowly
- Run on faster VMs (not sure if we can)

https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=caf13ef7ff7c&showall=1
https://treeherder.mozilla.org/ui/logviewer.html#?job_id=2757778&repo=try

Comment 81

[Jonathan Griffin (:jgriffin)]

Comment on attachment 8512949 [details] [diff] [review]
m-c] patch to run mulet reftest: 1) add --mulet flag 2) allow external connections, 3) do not add gaia prefs and 4) add sleep

Review of attachment 8512949 [details] [diff] [review]:
-----------------------------------------------------------------

::: layout/tools/reftest/b2g_desktop.py
@@ +45,5 @@
>          self.marionette.start_session()
> +        if self.build_type == "mulet":
> +            # XXX: bug 1043699 - unlocking takes too long on EC2
> +            import time
> +            time.sleep(15)

We already know how to get around this problem, see http://dxr.mozilla.org/mozilla-central/source/testing/web-platform/harness/wptrunner/browsers/b2g.py#236

Comment 82

[Armen [:armenzg]]

Attachment: [checked-in] [m-c] patch to run mulet reftest: 1) add --mulet flag, 2) do not add gaia prefs and 3) wait for screen

Pushed to try:
https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=787ab3d5bb8e

I will append a patch for buildbot-configs to chunkify.

I added this to the patch to test chunking:
--- a/testing/config/mozharness/linux_mulet_config.py
+++ b/testing/config/mozharness/linux_mulet_config.py
@@ -1,13 +1,11 @@
 # This is used by mozharness' mulet_unittest.py
 config = {
     # testsuite options
     "reftest_options": [
         "--mulet", "--profile=%(gaia_profile)s",
-        "--appname=%(application)s", "%(test_manifest)s",
-        "--this-chunk", "1",
-        "--total-chunk", "6"
+        "--appname=%(application)s", "%(test_manifest)s"
     ],
     "run_file_names": {
         "reftest": "runreftestb2g.py",
     },
 }

Comment 83

[Armen [:armenzg]]

Attachment: [b-c] add 4 chunks for mulet reftests

Comment 84

[Armen [:armenzg]]

Comment on attachment 8514467 [details] [diff] [review]
[checked-in] [m-c] patch to run mulet reftest: 1) add --mulet flag, 2) do not add gaia prefs and 3) wait for screen

I want to land this once I get a review for the recent changes I made.

For 6 chunks we get a 33 minute run for the push I made.

I will update my b-c configs to have 6 chunks instead of 4.

Chunk 1/6:
https://treeherder.mozilla.org/ui/logviewer.html#?job_id=2833982&repo=try

Comment 85

[Armen [:armenzg]]

Attachment: [checked-in] [b-c] add 6 chunks for mulet reftests

Comment 86

[Jonathan Griffin (:jgriffin)]

Comment on attachment 8514467 [details] [diff] [review]
[checked-in] [m-c] patch to run mulet reftest: 1) add --mulet flag, 2) do not add gaia prefs and 3) wait for screen

Review of attachment 8514467 [details] [diff] [review]:
-----------------------------------------------------------------

Lgtm, thanks!

Comment 87

[Jordan Lund (:jlund)]

Comment on attachment 8514959 [details] [diff] [review]
[checked-in] [b-c] add 6 chunks for mulet reftests

Review of attachment 8514959 [details] [diff] [review]:
-----------------------------------------------------------------

lgtm :)

I'm assuming we must wait for https://bug1043699.bugzilla.mozilla.org/attachment.cgi?id=8514467 to land?

do we need a tbpl patch to recognize the chunking?

Comment 88

[Jordan Lund (:jlund)]

Comment on attachment 8514470 [details] [diff] [review]
[b-c] add 4 chunks for mulet reftests

Review of attachment 8514470 [details] [diff] [review]:
-----------------------------------------------------------------

I'm assuming this is obsolete now based of newer patch.

If you going to iterate by landing 4 chunks and then 6 chunks later and you need a review, feel free to r+ this.

Comment 89

[Armen [:armenzg]]

Comment on attachment 8500612 [details] [diff] [review]
[checked-in] [buildbot-configs] enable mulet reftests on Cedar

Checked-in a while ago.

Comment 90

[Armen [:armenzg]]

Comment on attachment 8514959 [details] [diff] [review]
[checked-in] [b-c] add 6 chunks for mulet reftests

Waiting for a reconfig.
http://hg.mozilla.org/build/buildbot-configs/rev/f6d6c9302c55

Comment 91

[Armen [:armenzg]]

We don't need for the mulet change to land first to run the reconfig.
However, I will be landing before EOD assuming I see no new surprises in my last try test:
https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=5258a11ecc3c

Comment 92

[Armen [:armenzg]]

Comment on attachment 8514467 [details] [diff] [review]
[checked-in] [m-c] patch to run mulet reftest: 1) add --mulet flag, 2) do not add gaia prefs and 3) wait for screen

https://hg.mozilla.org/integration/mozilla-inbound/rev/6921bd616ff1

Comment 93

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/6921bd616ff1

Comment 94

[Armen [:armenzg]]

Comment on attachment 8514467 [details] [diff] [review]
[checked-in] [m-c] patch to run mulet reftest: 1) add --mulet flag, 2) do not add gaia prefs and 3) wait for screen

I landed the wrong patch.
I will land the right one tomorrow after I get the right thing pushed to try.

Comment 95

[Ed Morley [:emorley]]

Comment on attachment 8514467 [details] [diff] [review]
[checked-in] [m-c] patch to run mulet reftest: 1) add --mulet flag, 2) do not add gaia prefs and 3) wait for screen

(This patch no longer allows external connections, so removing the mention of that)

Comment 96

[Armen [:armenzg]]

Thanks for updating the comment for the patch.

Pushing the right patch:
https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=e55985ad7e90

Comment 97

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/ec12c4e4bcd3

Comment 98

[Armen [:armenzg]]

Comment on attachment 8514467 [details] [diff] [review]
[checked-in] [m-c] patch to run mulet reftest: 1) add --mulet flag, 2) do not add gaia prefs and 3) wait for screen

https://hg.mozilla.org/integration/mozilla-inbound/rev/dfd2eb97d840

Comment 99

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/dfd2eb97d840

Comment 100

[Armen [:armenzg]]

Unfortunately something has changed since comment 84 (2014-10-30) [1][2]

Who knows how we can disable the checking for updates? It is breaking the rule of hitting the network. [3]

I can reproduce the issue locally. The Mulet starts up, however, it aborts soon after. [4]

Allowing non-local connections makes it work again (even though this is not what we want to do).
I've pushed it to try so we can know where we stand once we fix the checking for updates.
https://tbpl.mozilla.org/?tree=Try&rev=be3fb02d83bd&show_all=1

[1]
Results from comment 84:
https://treeherder.mozilla.org/ui/logviewer.html#?job_id=2833982&repo=try
13:55:45     INFO - TinderboxPrint: reftest<br/>1733/<em class="testfail">120</em>/78
Log:
http://ftp.mozilla.org/pub/mozilla.org/b2g/try-builds/[e-mail]-787ab3d5bb8e/try-linux64-mulet/try_ubuntu64_vm-mulet_test-reftest-bm113-tests1-linux64-build55.txt.gz

[2] Chunkified results:
https://tbpl.mozilla.org/?tree=Try&rev=a89990bc0b83&show_all=1

[3]
09:18:10     INFO -  UpdatePrompt: Forcing update check
09:18:10     INFO -  *** AUS:SVC Checker: checkForUpdates, force: true
09:18:10     INFO -  *** AUS:SVC getLocale - getting locale from file: resource://gre/update.locale, locale: en-US
09:18:10     INFO -  *** AUS:SVC Checker:getUpdateURL - update URL: https://aus4.mozilla.org/update/3/Firefox/36.0a1/20141104131517/%PRODUCT_DEVICE%/en-US/default/Linux%203.13.0-39-generic%20(GTK%202.24.23)/default/default/update.xml?force=1
09:18:10     INFO -  *** AUS:SVC gCanCheckForUpdates - able to check for updates
09:18:10     INFO -  *** AUS:SVC recordInHealthReport - updateCheckStart - 0
09:18:10     INFO -  *** AUS:SVC Checker:checkForUpdates - sending request to: https://aus4.mozilla.org/update/3/Firefox/36.0a1/20141104131517/%PRODUCT_DEVICE%/en-US/default/Linux%203.13.0-39-generic%20(GTK%202.24.23)/default/default/update.xml?force=1
09:18:10     INFO -  FATAL ERROR: Non-local network connections are disabled and a connection attempt to aus4.mozilla.org (63.245.217.219) was made.
09:18:10     INFO -  You should only access hostnames available via the test networking proxy (if running mochitests) or from a test-specific httpd.js server (if running xpcshell tests). Browser services should be disabled or redirected to a local server.

[4]
NOTE: If you already have a gaia repo you can point to it like I do

* Installers from 2014-11-05:
python scripts/mulet_unittest.py --cfg b2g/generic_config.py --cfg b2g/mulet_config.py --test-suite reftest --test-manifest tests/layout/reftests/reftest.list  --blob-upload-branch try --download-symbols ondemand --installer-url http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-a89990bc0b83/try-linux64-mulet/firefox-36.0a1.en-US.linux-x86_64.tar.bz2 --test-url http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-a89990bc0b83/try-linux64-mulet/firefox-36.0a1.en-US.linux-x86_64.tests.zip --gaia-repo ~/repos/branches/gaia-central  --cfg developer_config.py --total-chunks 6 --this-chunk 1

Comment 101

[Armen [:armenzg]]

ochameau: who knows how we can disable the checking for updates? It is breaking the rule of hitting the network. [3]

Side note, we need to watch for this change that *could* be affecting mulet reftests (probably not related):
https://bugzilla.mozilla.org/show_bug.cgi?id=1093691#c19

Comment 102

[Jonathan Griffin (:jgriffin)]

(In reply to Armen Zambrano - Automation & Tools Engineer (:armenzg) from comment #101)
> ochameau: who knows how we can disable the checking for updates? It is
> breaking the rule of hitting the network. [3]
> 
> Side note, we need to watch for this change that *could* be affecting mulet
> reftests (probably not related):
> https://bugzilla.mozilla.org/show_bug.cgi?id=1093691#c19

Can you help identify someone to help with this, gwagner?

Comment 103

[Armen [:armenzg]]

I have filed two bugs (bug 1043699 and bug 1094369) which (IIUC) need dev attention.

I've also pushed a try push with 25 chunks instead of 6 since we're timing out after two hours.
I wonder what changed since in comment 84 we were having 30 minutes run.
https://tbpl.mozilla.org/?tree=Try&rev=1c38b6313fa3&show_all=1
I think we have to wait for the dep bugs to be fixed first before doing the estimate but it might be good to have a data point for reference.

In one sense I think this bug is done wrt to making the harness work, however, I will wait to hear from jgriffing/ochameau as to if more help is needed from me.

Comment 104

[Gregor Wagner [:gwagner]]

Alex, can you help out here?

Comment 105

[Armen [:armenzg]]

I have landed a change on Cedar to allow seeing the jobs run by disabling the network block [1]
Once we have a fix for bug 1094362 we can back the change out on Cedar.

Once we fix the reftests locally (bug 1094369) please let me know so we can determine how many chunks we need (as of _now_ I think we will need 15 or more) [2].

[1] http://hg.mozilla.org/projects/cedar/rev/bfe35dbdef31
https://tbpl.mozilla.org/?tree=Cedar&jobname=Mulet&show_all=1

[2]
https://tbpl.mozilla.org/?tree=Try&rev=1c38b6313fa3&show_all=1

Comment 106

[:gerard-majax]

Armen, is it possible you are being hit by the same thing that made Mnw perma fail ?
Patch for bug 1092941 should help if it's the case.

Or I misread why I'm being asked for help here. If that's the case, please forgive and tell me what's wrong and how I can try to help.

Comment 107

[Armen [:armenzg]]

(In reply to Alexandre LISSY :gerard-majax from comment #106)
> Armen, is it possible you are being hit by the same thing that made Mnw
> perma fail ?
> Patch for bug 1092941 should help if it's the case.
> 
> Or I misread why I'm being asked for help here. If that's the case, please
> forgive and tell me what's wrong and how I can try to help.

After chatting on IRC, it seems that for Mulet mochitest we disabled the network block [1] (bug 1066044).
We should undo that.

We first need to disable the updates, we can then look at which failures we have.

[1] https://hg.mozilla.org/mozilla-central/rev/3538768a23f4#l7.1

Comment 108

[Alexandre Poirot [:ochameau]]

It's really unclear why mulet ends up having updates.
We try really hard to disable it in multiple places, and especially, here, for reftest:
  http://mxr.mozilla.org/mozilla-central/source/layout/tools/reftest/reftest-preferences.js#17
Aren't we using these prefs when running reftests on mulet??

Comment 109

[Armen [:armenzg]]

The prefs are being ignored since we force the update.

Let's move the check for update issue to bug 1094362.

Comment 110

[Armen [:armenzg]]

The harness work is done in here.

To make the tests go green please follow up in bug 1094362 and bug 1094369.

Comment 111

[Firefox Bug Husbandry Bot]

Removing leave-open keyword from resolved bugs, per :sylvestre.