Skip to content
Snippets Groups Projects

tests: Mark a number of test failures as buggy on macOS before Sequoia.

Merged Giovanni Mascellani requested to merge giomasce/vkd3d:vesuvio into master

The phylosophy of this MR is that catching regressions is more valuable than investigating each single failure. I've already investigated many, but these are the hardest ones, they don't even happen on Sequoia, and we already missed a number of macOS regressions because the CI script was allowed to fail and nobody cared to notice. I plan to keep working on the DXC tests anyway.

This MR will likely fail anyway, because given how the CI script works the allow_failure flag retroactively applies also to the first commit. However, once it's in, it should do the right thing. Ah, I had forgotten that also the environment variables apply retroactively, so no failure at all! :-)

Edited by Giovanni Mascellani

Merge request reports

Loading
Loading

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
  • Giovanni Mascellani changed the description

    changed the description

  • BTW, if we want to keep DXC tests we can do an arrangement like we do for Windows tests.

  • added 1 commit

    • 4186c770 - tests: Mark a todo as resolved on MoltenVK 1.2.12.

    Compare with previous version

  • added 1 commit

    • 2221e7bc - tests: Mark a todo as resolved on MoltenVK 1.2.12.

    Compare with previous version

    • The phylosophy of this MR is that catching regressions is more valuable than investigating each single failure. I've already investigated many, but these are the hardest ones, they don't even happen on Sequoia, and we already missed a number of macOS regressions because the CI script was allowed to fail and nobody cared to notice. I plan to keep working on the DXC tests anyway.

      I'm not completely sure which regressions you're referring to (the main recent issue I'm aware of is the one fixed by 9f09ac68; arguably the larger issue there is the lack of test coverage for effects), but note that I manually run the macOS tests locally before merging commits. Unfortunately I'll have to keep doing that even after this MR, because the CI isn't quite capable enough to run the MSL tests. My macOS setup does have some difference with the CI; it's running Sequoia, and I'm compiling with gcc.

      The latter is also where this MR breaks down for me somewhat. __builtin_available() is not available on gcc, and causes compilation to fail here.

      BTW, if we want to keep DXC tests we can do an arrangement like we do for Windows tests.

      There doesn't seem much of a point until those tests start passing. In principle it's nice to have the infrastructure for running those tests in place already to make it easier for someone else to pick up, but in practice that doesn't really seem to happen, and the changes are pretty minimal in this case anyway.

    • I'm not completely sure which regressions you're referring to (the main recent issue I'm aware of is the one fixed by 9f09ac68; arguably the larger issue there is the lack of test coverage for effects), but note that I manually run the macOS tests locally before merging commits. Unfortunately I'll have to keep doing that even after this MR, because the CI isn't quite capable enough to run the MSL tests. My macOS setup does have some difference with the CI; it's running Sequoia, and I'm compiling with gcc.

      I wouldn't be able to point point that more precisely without taking some time, but I'm pretty other events like that happened in the past. But even if I'm wrong, I'm not sure I understand: are you argueing against my proposal, or is that just a side comment?

      I agree that you'll have to keep running tests locally, and I'll keep doing so too with my scripts, even if post merge. But I would say that having a light going red automatically, even if just in a subset of the cases we can detect, is better than nothing. It saves you some manual work and the MR submitter a round trip.

      The latter is also where this MR breaks down for me somewhat. __builtin_available() is not available on gcc, and causes compilation to fail here.

      Ah, I didn't know you use gcc. Hopefully the latest proposal is more compatible.

      There doesn't seem much of a point until those tests start passing. In principle it's nice to have the infrastructure for running those tests in place already to make it easier for someone else to pick up, but in practice that doesn't really seem to happen, and the changes are pretty minimal in this case anyway.

      Well, the infrastructure is there, now just commented out. And I guess that "someone" is eventually going to be me, therefore not being "someone else", but still there is hope that eventually the infrastructure will be in good used.

    • Please register or sign in to reply
  • added 3 commits

    • 9a9e9a5a - tests: Mark a number of test failures as buggy on macOS before Sequoia.
    • 0807f70b - tests: Disable testing with DXC on macOS.
    • fe008db7 - tests: Mark a todo as resolved on MoltenVK 1.2.12.

    Compare with previous version

    • I'm not completely sure which regressions you're referring to (the main recent issue I'm aware of is the one fixed by 9f09ac68; arguably the larger issue there is the lack of test coverage for effects), but note that I manually run the macOS tests locally before merging commits. Unfortunately I'll have to keep doing that even after this MR, because the CI isn't quite capable enough to run the MSL tests. My macOS setup does have some difference with the CI; it's running Sequoia, and I'm compiling with gcc.

      I wouldn't be able to point point that more precisely without taking some time, but I'm pretty other events like that happened in the past. But even if I'm wrong, I'm not sure I understand: are you argueing against my proposal, or is that just a side comment?

      I'm mostly just trying to figure out whether I somehow missed a bunch of regressions since I started running the tests on macOS (i.e., November last year, roughly), and should be adjusting my process, or whether these are from before that time, and things are essentially fine on my side.

      I don't have an issue with this MR as such; quite the contrary, I'm grateful this is happening.

      There doesn't seem much of a point until those tests start passing. In principle it's nice to have the infrastructure for running those tests in place already to make it easier for someone else to pick up, but in practice that doesn't really seem to happen, and the changes are pretty minimal in this case anyway.

      Well, the infrastructure is there, now just commented out. And I guess that "someone" is eventually going to be me, therefore not being "someone else", but still there is hope that eventually the infrastructure will be in good used.

      Right, so no need for a separate macos-dxc job at this point. :)

    • I'm mostly just trying to figure out whether I somehow missed a bunch of regressions since I started running the tests on macOS (i.e., November last year, roughly), and should be adjusting my process, or whether these are from before that time, and things are essentially fine on my side.

      Ah, got it. Well, it might definitely have been before then, I don't remember.

    • Please register or sign in to reply
  • Henri Verbeet approved this merge request

    approved this merge request

  • Henri Verbeet added 27 commits

    added 27 commits

    • fe008db7...25400819 - 24 commits from branch wine:master
    • aab8ba02 - tests: Mark a number of test failures as buggy on macOS before Sequoia.
    • 653e4f47 - ci: Disable testing with DXC on macOS.
    • 7b4a29da - tests: Mark a todo as resolved on MoltenVK 1.2.12.

    Compare with previous version

  • merged

Please register or sign in to reply
Loading