dEQP

These steps are specifically for testing SwiftShader's OpenGL ES 3.0 implementation using dEQP on Windows (steps for Linux below the Windows instructions).

Prerequisites

  1. Install the latest Python 3
  2. Install Visual Studio
  3. Install CMake
  4. Install Go
  5. Install MinGW-W64
  • Select ‘x86_64’ as Architecture during setup
  1. Install Git
  2. Install Android Studio
  3. Run Android Studio and install Android SDK.
  4. Set environment variables: Config Panel -> System and Security -> System -> Advanced system settigns -> Environment Variables

  • Add <path to python> to your PATH environment variable

  • Add <path to MinGW-W64>\bin to your PATH environment variable

  • Add <path to adb> to your PATH environment variable

    Note: abd is in the Android SDK, typically in C:\Users\<username>\AppData\Local\Android\sdk\platform-tools

  1. (Optional) Install TortoiseGit

Getting the Code

  1. Get dEQP (either in ‘cmd’ or by using TortoiseGit):

    git clone https://github.com/KhronosGroup/VK-GL-CTS

    You may wish to check out a stable vulkan-cts-* branch.

  2. Get dEQP's dependencies. In your dEQP root directory, open ‘cmd’ and run:

    python3 external\fetch_sources.py

  3. Get Cherry (either in ‘cmd’ or by using TortoiseGit):

    git clone https://android.googlesource.com/platform/external/cherry

  4. Set environment variable (see point 9):

    Add new variable GOPATH=‘<path to cherry>’

Building the code

  1. Build dEQP's Visual Studio files using the CMake GUI, or, in the dEQP root dir, run:

    mkdir build
cd build
cmake ..

    Note: don't call ‘cmake .’ directly in the root directory. It will make things fails later on. If you do, simply erase the files created by CMake and follow the steps above.

  2. Build dEQP:

    Open <path to dEQP>\build\dEQP-Core-default.sln in Visual Studio and Build Solution

    Note: Choose a ‘Release’ build, unless you really mean to debug dEQP

  3. Generate test cases:

    mkdir <path to cherry>\data
cd <path to dEQP>
python3 scripts\build_caselists.py <path to cherry>\data

    Note: you need to run python3 scripts\build_caselists.py <path to cherry>\data every time you update dEQP.

Preparing the server

  1. Edit <path to cherry>\cherry\data.go
  • Search for ../candy-build/deqp-wgl and replace that by <path to deqp>/build
  • Just above, add an option to CommandLine: --deqp-gl-context-type=egl
  • Just below, modify the BinaryPath from ‘Debug’ to ‘Release’ if you did a Release build at step 17
  • Remove --deqp-watchdog=enable to avoid timeouts during debugging.

Testing OpenGL ES

  1. a) Assuming you already built SwiftShader in the build folder, copy these two files:

    libEGL.dll
    libGLESv2.dll

    From:

    <path to SwiftShader>\build\Release_x64 or
    <path to SwiftShader>\build\Debug_x64

    To:

    <path to dEQP>\build\modules\gles3\Release (Again, assuming you did a Release build at step 17)

Testing Vulkan

  1. b) Assuming you already built SwiftShader, copy and rename this file:

    <path to SwiftShader>\build\Release_x64\vk_swiftshader.dll or
    <path to SwiftShader>\build\Debug_x64\vk_swiftshader.dll

    To:

    <path to dEQP>\build\external\vulkancts\modules\vulkan\vulkan-1.dll

    This will cause dEQP to load SwiftShader's Vulkan implementatin directly, without going through a system-provided loader library or any layers.

    To use SwiftShader as an Installable Client Driver (ICD) instead:

    • Edit environment variables:
      • Define VK_ICD_FILENAMES to <path to SwiftShader>\src\Vulkan\vk_swiftshader_icd.json
    • If the location of vk_swiftshader.dll you're using is different than the one specified in src\Vulkan\vk_swiftshader_icd.json, modify it to point to the vk_swiftshader.dll file you want to use.

Running the tests

  1. Start the test server. Go to <path to cherry> and run:

    go run server.go

  2. Open your favorite browser and navigate to localhost:8080

    Get Started -> Choose Device ‘localhost’ -> Select Tests ‘dEQP-GLES3’ -> Execute tests!

Mustpass sets

dEQP contains more tests than what is expected to pass by a conformant implementation (e.g. some tests are considered too strict, or assume certain undefined behavior). The android\cts\master\gles3-master.txt text file which can be loaded in Cherry's ‘Test sets’ tab to only run the latest tests expected to pass by certified Android devices.

Linux

The Linux process is similar to Windows. However it doesn't use Release or Debug variants, paths use forward slashes, and it uses shared object files instead of DLLs.

  1. Install the latest Python 3

  2. Install GCC and Make. In a terminal, run:

    sudo apt-get install gcc make

  3. Install CMake

  4. Install Go

  5. Install Git. In a terminal, run:

    sudo apt-get install git

  6. Download the Vulkan SDK and unpack it into a location you like.

Getting the Code

  1. Get Swiftshader. In a terminal, go to the location you want to keep Swiftshader, and run:

    git clone https://swiftshader.googlesource.com/SwiftShader && (cd SwiftShader && curl -Lo `git rev-parse --git-dir`/hooks/commit-msg https://gerrit-review.googlesource.com/tools/hooks/commit-msg ; chmod +x `git rev-parse --git-dir`/hooks/commit-msg)

    This will also install the commit hooks you need for committing to SwiftShader.

  2. Get dEQP:

    git clone sso://googleplex-android/platform/external/deqp

  3. Get dEQP's dependencies. In your dEQP root directory, run:

    python3 external/fetch_sources.py

  4. Get Cherry, similar to step 8:

    git clone https://android.googlesource.com/platform/external/cherry

  5. Set environment variable. Open ~/.bashrc in your preferred editor and add the following line:

    GOPATH=‘<path to cherry>’

Building the code

  1. Build Swiftshader. In the Swiftshader root dir, run:

    cd build
cmake ..
make --jobs=$(nproc)

  2. Set your environment variables. In the terminal in which you'll be building dEQP, run the following commands:

    export LD_LIBRARY_PATH="<Vulkan SDK location>/x86_64/lib:$LD_LIBRARY_PATH"
export LD_LIBRARY_PATH="<Swiftshader location>/build:$LD_LIBRARY_PATH"

    It‘s important that you perform this step before you build dEQP in the next step. CMake will search for library files in LD_LIBRARY_PATH. If it cannot discover Swiftshader’s libEGL and libGLESv2 shared object files, then CMake will default to using your system's libEGL.so and libGLESv2.so files.

  3. Build dEQP. In the dEQP root dir, run:

    mkdir build
cd build
cmake ..
make --jobs=$(nproc)

    Also: don't call ‘cmake .’ directly in the root directory. It will make things fails later on. If you do, simply erase the files created by CMake and follow the steps above.

  4. Generate test cases:

    mkdir <path to cherry>/data
cd <path to dEQP>
python3 scripts/build_caselists.py <path to cherry>/data

    Note: you need to run python3 scripts/build_caselists.py <path to cherry>/data every time you update dEQP.

Preparing the server

  1. Edit <path to cherry>/cherry/data.go
  • Search for “.exe” and remove all instances.
  • Search for ../candy-build/deqp-wgl/execserver/Release and replace that by <path to deqp>/build/execserver/execserver
  • Just above, add an option to CommandLine: --deqp-gl-context-type=egl
  • Just below, remove ‘Debug/’ from the BinaryPath.
  • Just one more line below, replace ../candy-build/deqp-wgl/ with <path to deqp>/build/modules/${TestPackageDir}.
  • Remove --deqp-watchdog=enable to avoid timeouts during debugging.

Testing OpenGL ES

  1. a) Assuming you setup the LD_LIBRARY_PATH environment variable prior to running CMake in the dEQP build directory, you're all set.

Testing Vulkan

  1. b) Use SwiftShader as an Installable Client Driver (ICD). Add the following line to your ~/.bashrc:

    export VK_ICD_FILENAMES="<path to SwiftShader>/build/Linux/vk_swiftshader_icd.json"

    Then run source ~/.bashrc the terminal(s) you'll be running tests from.

Running the tests

  1. Start the test server. Go to <path to cherry> and run:

    go run server.go

  2. Open your favorite browser and navigate to localhost:8080

    Get Started -> Choose Device ‘localhost’ -> Select Tests ‘dEQP-GLES3’ -> Execute tests!

  3. To make sure that you‘re running SwiftShader’s drivers, select only the dEQP-GLES3->info->vendor and dEQP-VK->info->platform tests. In the next window, click on these tests in the left pane. If you see Google inc for the GLES3 test and your Linux machine in the VK test, then you've set your suite up properly.

  4. If you want to run Vulkan tests in the command line, go to the build directory in dEQP root. Then run the following command:

    external/vulkanacts/modules/vulkan/deqp-vk

    You can also run individual tests with:

    external/vulkanacts/modules/vulkan/deqp-vk --deqp-case=<test name>

    And you can find a list of the test names in <Swiftshader root>/tests/regres/testlists/vk-master.txt However, deqp-vk will cease upon the first failure. It‘s recommended that you use cherry for your testing needs unless you know what you’re doing.

  5. To check that you're running SwiftShader in cherry, start the server

Mustpass sets

dEQP contains more tests than what is expected to pass by a conformant implementation (e.g. some tests are considered too strict, or assume certain undefined behavior). The android\cts\master\gles3-master.txt text file which can be loaded in Cherry's ‘Test sets’ tab to only run the latest tests expected to pass by certified Android devices.