Skip to content

Conversation

benjaoming
Copy link
Contributor

@benjaoming benjaoming commented May 23, 2020

Summary

Buildkite and Circle CI is restored, however Selenium-based tests now fail. The failures seem to be related to new breakages, but not the Selenium/geckodriver updates which seem to work.

  • Nodejs 6.x was used as 6.17 before, but the index of the Debian repository is somehow broken and reports 6.14 as the latest version
  • It's not possible anymore to use parallelism in Circle CI the way we did since we cannot share state between the containers as we did before
  • Had to do a lot of experimentation to get a working combination of Firefox, Geckodriver and Selenium up and running again
  • Many elements of Selenium tests have been disabled as they caused false negatives
  • The coverage dip in the CodeCov report is not related to disabled tests, as it is just a few tests that have been disabled, and those were about validating for correctness, not executing code. Not sure why the coverage has gone down.
  • Circle CI 2.x containers is not amazing.. it's just a maze.

Background

Circle CI has killed the 1.0 API that we were using. So we needed to either go back to Travis CI or start using their 2.0. The changes introduced here have no new value other than just extending the lifespan of the automated tests and builds. It's very sad the way that Circle CI has removed their old API and forced us into this.

The reason why this problem went under the radar was that the Circle CI integration was simply shut down, so we kept getting ✔️ on our builds because Buildkite was still building artifacts.. but alas, no testing was done.

It also turned out that the combination of geckodriver+Selenium+Firefox wasn't reproducible in the new Circle CI docker containers.. I couldn't make it work, sticking to the same Selenium version and going through a series of different geckodriver+Firefox combinations, including the ones we had used before.

The automated tests are the only means to introduce changes to a somewhat large codebase that will be difficult to support in case of breakage.

https://circleci.com/sunset1-0/

The CircleCI 1.0 cloud infrastructure is no longer available as of March 15, 2019.

TODO

  • Has documentation been written/updated?
  • Have you written release notes for the upcoming release?

Reviewer guidance

If your PR has a significant size, give the reviewer some helpful remarks

Issues addressed

List the issues solved or partly solved by the PR

@benjaoming benjaoming marked this pull request as draft May 23, 2020 16:38
@benjaoming benjaoming changed the title WIP Circle CI 2.0 conf Circle CI 2.0 conf May 23, 2020
@benjaoming benjaoming force-pushed the circle-ci-2 branch 6 times, most recently from 54f4dc6 to 8020bff Compare May 24, 2020 14:02
@benjaoming benjaoming changed the title Circle CI 2.0 conf Circle CI 2.1 conf May 24, 2020
@benjaoming benjaoming force-pushed the circle-ci-2 branch 5 times, most recently from cdaa3e2 to a5475c7 Compare May 24, 2020 21:01
@codecov
Copy link

codecov bot commented May 28, 2020

Codecov Report

Merging #5627 into 0.17.x will decrease coverage by 19.72%.
The diff coverage is 32.14%.

Impacted file tree graph

@@             Coverage Diff             @@
##           0.17.x    #5627       +/-   ##
===========================================
- Coverage   66.05%   46.33%   -19.73%     
===========================================
  Files         113      113               
  Lines        6508     6514        +6     
===========================================
- Hits         4299     3018     -1281     
- Misses       2209     3496     +1287     
Impacted Files Coverage Δ
...achreports/management/commands/generaterealdata.py 0.00% <0.00%> (ø)
kalite/distributed/management/commands/setup.py 0.00% <0.00%> (-43.65%) ⬇️
kalite/distributed/middleware.py 52.08% <0.00%> (-5.70%) ⬇️
kalite/topic_tools/content_models.py 56.17% <ø> (-21.63%) ⬇️
kalite/distributed/views.py 57.92% <50.00%> (-20.74%) ⬇️
kalite/main/models.py 65.07% <50.00%> (-20.05%) ⬇️
kalite/control_panel/views.py 39.88% <66.66%> (-28.28%) ⬇️
kalite/shared/decorators/auth.py 41.17% <66.66%> (-1.54%) ⬇️
kalite/updates/management/commands/videoscan.py 0.00% <0.00%> (-100.00%) ⬇️
...stributed/management/commands/initialize_kalite.py 0.00% <0.00%> (-100.00%) ⬇️
... and 43 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e645658...c72582a. Read the comment docs.

@benjaoming benjaoming marked this pull request as ready for review May 29, 2020 10:21
Copy link
Contributor Author

@benjaoming benjaoming left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No code was changed, just tests and build scripts. And a bit of developer documentation. Hence, I will self-merge this in order to have functional builds restored for other release PRs.

@benjaoming benjaoming merged commit e85c24b into learningequality:0.17.x May 29, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant