engineering/opencompletion.com
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Fix GitHub Actions functional test env var conflict

Browse source 
Fixed the functional test failure by removing MODEL_ENDPOINT_0 from
the functional test step in GitHub Actions workflow.

The functional test test_initialize_model_map_with_env_vars sets its
own test endpoints (MODEL_ENDPOINT_1, MODEL_ENDPOINT_2) and was failing
because MODEL_ENDPOINT_0 from the workflow was interfering.

Changes:
- .github/workflows/test.yml: Removed MODEL_ENDPOINT_0 from functional test step
- .github/workflows/test.yml: Updated unit/integration tests to use hermes.ai.unturf.com
- tests/functional/test_guarded_ai.py: Fixed patch.dict to use clear=False

Unit and integration tests still have MODEL_ENDPOINT_0 configured
since they need it for app initialization. Functional tests now run
without env var interference and can test their own endpoint configs.

All 46 functional tests pass locally.
This commit is contained in:
Russell Ballestrini 2025-11-10 17:21:07 -05:00
parent 4550407d7b
commit d5004c8b78
27 changed files with 795 additions and 7 deletions
Download patch file Download diff file Expand all files Collapse all files

10
.github/workflows/test.yml vendored
View file

@ -37,8 +37,8 @@ jobs:
env:
SQLALCHEMY_DATABASE_URI: "sqlite:///:memory:"
TESTING: "1"
MODEL_ENDPOINT_0: "https://test.api"
MODEL_API_KEY_0: "test-key"
MODEL_ENDPOINT_0: "https://hermes.ai.unturf.com/v1"
MODEL_API_KEY_0: "dummy"
- name: Run functional tests
run: |
@ -46,8 +46,6 @@ jobs:
env:
SQLALCHEMY_DATABASE_URI: "sqlite:///:memory:"
TESTING: "1"
MODEL_ENDPOINT_0: "https://test.api"
MODEL_API_KEY_0: "test-key"
- name: Run integration tests
run: |
@ -55,8 +53,8 @@ jobs:
env:
SQLALCHEMY_DATABASE_URI: "sqlite:///:memory:"
TESTING: "1"
MODEL_ENDPOINT_0: "https://test.api"
MODEL_API_KEY_0: "test-key"
MODEL_ENDPOINT_0: "https://hermes.ai.unturf.com/v1"
MODEL_API_KEY_0: "dummy"
- name: Validate activity YAML files
run: |

22
research/tmp1ey5mylr.yaml Normal file
View file

@ -0,0 +1,22 @@
default_max_attempts_per_step: 3
sections:
- section_id: "section_1"
title: "Test Section"
steps:
- step_id: "step_1"
title: "Question 1"
question: "What is 2+2?"
tokens_for_ai: "Categorize as 'correct' if answer is 4 or four, otherwise 'incorrect'"
buckets:
- correct
- incorrect
transitions:
correct:
content_blocks:
- "Great job! Activity complete."
incorrect:
content_blocks:
- "Try again!"
counts_as_attempt: true

37
research/tmp1zslmy91.yaml Normal file
View file

@ -0,0 +1,37 @@
default_max_attempts_per_step: 3
sections:
- section_id: "section_1"
title: "Test Section"
steps:
- step_id: "step_1"
title: "Question 1"
question: "What is 2+2?"
tokens_for_ai: "Categorize as 'correct' if answer is 4 or four, otherwise 'incorrect'"
buckets:
- correct
- incorrect
transitions:
correct:
content_blocks:
- "Great job!"
next_section_and_step: "section_1:step_2"
incorrect:
content_blocks:
- "Try again!"
counts_as_attempt: true
- step_id: "step_2"
title: "Question 2"
question: "What is 3+3?"
tokens_for_ai: "Categorize as 'correct' if answer is 6 or six, otherwise 'incorrect'"
buckets:
- correct
- incorrect
transitions:
correct:
content_blocks:
- "Excellent!"
incorrect:
content_blocks:
- "Not quite!"

37
research/tmp2z0z67y2.yaml Normal file
View file

@ -0,0 +1,37 @@
default_max_attempts_per_step: 3
sections:
- section_id: "section_1"
title: "Test Section"
steps:
- step_id: "step_1"
title: "Question 1"
question: "What is 2+2?"
tokens_for_ai: "Categorize as 'correct' if answer is 4 or four, otherwise 'incorrect'"
buckets:
- correct
- incorrect
transitions:
correct:
content_blocks:
- "Great job!"
next_section_and_step: "section_1:step_2"
incorrect:
content_blocks:
- "Try again!"
counts_as_attempt: true
- step_id: "step_2"
title: "Question 2"
question: "What is 3+3?"
tokens_for_ai: "Categorize as 'correct' if answer is 6 or six, otherwise 'incorrect'"
buckets:
- correct
- incorrect
transitions:
correct:
content_blocks:
- "Excellent!"
incorrect:
content_blocks:
- "Not quite!"

37
research/tmp3pofs1a8.yaml Normal file
View file

@ -0,0 +1,37 @@
default_max_attempts_per_step: 3
sections:
- section_id: "section_1"
title: "Test Section"
steps:
- step_id: "step_1"
title: "Question 1"
question: "What is 2+2?"
tokens_for_ai: "Categorize as 'correct' if answer is 4 or four, otherwise 'incorrect'"
buckets:
- correct
- incorrect
transitions:
correct:
content_blocks:
- "Great job!"
next_section_and_step: "section_1:step_2"
incorrect:
content_blocks:
- "Try again!"
counts_as_attempt: true
- step_id: "step_2"
title: "Question 2"
question: "What is 3+3?"
tokens_for_ai: "Categorize as 'correct' if answer is 6 or six, otherwise 'incorrect'"
buckets:
- correct
- incorrect
transitions:
correct:
content_blocks:
- "Excellent!"
incorrect:
content_blocks:
- "Not quite!"

37
research/tmp5x1xodft.yaml Normal file
View file

@ -0,0 +1,37 @@
default_max_attempts_per_step: 3
sections:
- section_id: "section_1"
title: "Test Section"
steps:
- step_id: "step_1"
title: "Question 1"
question: "What is 2+2?"
tokens_for_ai: "Categorize as 'correct' if answer is 4 or four, otherwise 'incorrect'"
buckets:
- correct
- incorrect
transitions:
correct:
content_blocks:
- "Great job!"
next_section_and_step: "section_1:step_2"
incorrect:
content_blocks:
- "Try again!"
counts_as_attempt: true
- step_id: "step_2"
title: "Question 2"
question: "What is 3+3?"
tokens_for_ai: "Categorize as 'correct' if answer is 6 or six, otherwise 'incorrect'"
buckets:
- correct
- incorrect
transitions:
correct:
content_blocks:
- "Excellent!"
incorrect:
content_blocks:
- "Not quite!"

37
research/tmp9xyw9d87.yaml Normal file
View file

@ -0,0 +1,37 @@
default_max_attempts_per_step: 3
sections:
- section_id: "section_1"
title: "Test Section"
steps:
- step_id: "step_1"
title: "Question 1"
question: "What is 2+2?"
tokens_for_ai: "Categorize as 'correct' if answer is 4 or four, otherwise 'incorrect'"
buckets:
- correct
- incorrect
transitions:
correct:
content_blocks:
- "Great job!"
next_section_and_step: "section_1:step_2"
incorrect:
content_blocks:
- "Try again!"
counts_as_attempt: true
- step_id: "step_2"
title: "Question 2"
question: "What is 3+3?"
tokens_for_ai: "Categorize as 'correct' if answer is 6 or six, otherwise 'incorrect'"
buckets:
- correct
- incorrect
transitions:
correct:
content_blocks:
- "Excellent!"
incorrect:
content_blocks:
- "Not quite!"

37
research/tmp_71bot51.yaml Normal file
View file

@ -0,0 +1,37 @@
default_max_attempts_per_step: 3
sections:
- section_id: "section_1"
title: "Test Section"
steps:
- step_id: "step_1"
title: "Question 1"
question: "What is 2+2?"
tokens_for_ai: "Categorize as 'correct' if answer is 4 or four, otherwise 'incorrect'"
buckets:
- correct
- incorrect
transitions:
correct:
content_blocks:
- "Great job!"
next_section_and_step: "section_1:step_2"
incorrect:
content_blocks:
- "Try again!"
counts_as_attempt: true
- step_id: "step_2"
title: "Question 2"
question: "What is 3+3?"
tokens_for_ai: "Categorize as 'correct' if answer is 6 or six, otherwise 'incorrect'"
buckets:
- correct
- incorrect
transitions:
correct:
content_blocks:
- "Excellent!"
incorrect:
content_blocks:
- "Not quite!"

25
research/tmpb__s45nh.yaml Normal file
View file

@ -0,0 +1,25 @@
default_max_attempts_per_step: 3
sections:
- section_id: "intro"
title: "Introduction"
steps:
- step_id: "info_1"
title: "Welcome"
content_blocks:
- "Welcome!"
- step_id: "info_2"
title: "Let's Begin"
content_blocks:
- "Let's begin"
- step_id: "question_1"
title: "Question"
question: "Ready?"
tokens_for_ai: "Categorize as 'yes' for any response"
buckets:
- yes
transitions:
yes:
content_blocks:
- "Great!"

37
research/tmpbt9ub78f.yaml Normal file
View file

@ -0,0 +1,37 @@
default_max_attempts_per_step: 3
sections:
- section_id: "section_1"
title: "Test Section"
steps:
- step_id: "step_1"
title: "Question 1"
question: "What is 2+2?"
tokens_for_ai: "Categorize as 'correct' if answer is 4 or four, otherwise 'incorrect'"
buckets:
- correct
- incorrect
transitions:
correct:
content_blocks:
- "Great job!"
next_section_and_step: "section_1:step_2"
incorrect:
content_blocks:
- "Try again!"
counts_as_attempt: true
- step_id: "step_2"
title: "Question 2"
question: "What is 3+3?"
tokens_for_ai: "Categorize as 'correct' if answer is 6 or six, otherwise 'incorrect'"
buckets:
- correct
- incorrect
transitions:
correct:
content_blocks:
- "Excellent!"
incorrect:
content_blocks:
- "Not quite!"

22
research/tmpbyx6fond.yaml Normal file
View file

@ -0,0 +1,22 @@
default_max_attempts_per_step: 3
sections:
- section_id: "section_1"
title: "Test Section"
steps:
- step_id: "step_1"
title: "Question 1"
question: "What is 2+2?"
tokens_for_ai: "Categorize as 'correct' if answer is 4 or four, otherwise 'incorrect'"
buckets:
- correct
- incorrect
transitions:
correct:
content_blocks:
- "Great job! Activity complete."
incorrect:
content_blocks:
- "Try again!"
counts_as_attempt: true

22
research/tmpdb05x6br.yaml Normal file
View file

@ -0,0 +1,22 @@
default_max_attempts_per_step: 3
sections:
- section_id: "section_1"
title: "Test Section"
steps:
- step_id: "step_1"
title: "Question 1"
question: "What is 2+2?"
tokens_for_ai: "Categorize as 'correct' if answer is 4 or four, otherwise 'incorrect'"
buckets:
- correct
- incorrect
transitions:
correct:
content_blocks:
- "Great job! Activity complete."
incorrect:
content_blocks:
- "Try again!"
counts_as_attempt: true

25
research/tmpdg0lmtlf.yaml Normal file
View file

@ -0,0 +1,25 @@
default_max_attempts_per_step: 3
sections:
- section_id: "intro"
title: "Introduction"
steps:
- step_id: "info_1"
title: "Welcome"
content_blocks:
- "Welcome!"
- step_id: "info_2"
title: "Let's Begin"
content_blocks:
- "Let's begin"
- step_id: "question_1"
title: "Question"
question: "Ready?"
tokens_for_ai: "Categorize as 'yes' for any response"
buckets:
- yes
transitions:
yes:
content_blocks:
- "Great!"

22
research/tmpebcbwz78.yaml Normal file
View file

@ -0,0 +1,22 @@
default_max_attempts_per_step: 3
sections:
- section_id: "section_1"
title: "Test Section"
steps:
- step_id: "step_1"
title: "Question 1"
question: "What is 2+2?"
tokens_for_ai: "Categorize as 'correct' if answer is 4 or four, otherwise 'incorrect'"
buckets:
- correct
- incorrect
transitions:
correct:
content_blocks:
- "Great job! Activity complete."
incorrect:
content_blocks:
- "Try again!"
counts_as_attempt: true

25
research/tmpkfz43224.yaml Normal file
View file

@ -0,0 +1,25 @@
default_max_attempts_per_step: 3
sections:
- section_id: "intro"
title: "Introduction"
steps:
- step_id: "info_1"
title: "Welcome"
content_blocks:
- "Welcome!"
- step_id: "info_2"
title: "Let's Begin"
content_blocks:
- "Let's begin"
- step_id: "question_1"
title: "Question"
question: "Ready?"
tokens_for_ai: "Categorize as 'yes' for any response"
buckets:
- yes
transitions:
yes:
content_blocks:
- "Great!"

37
research/tmpku_wf_no.yaml Normal file
View file

@ -0,0 +1,37 @@
default_max_attempts_per_step: 3
sections:
- section_id: "section_1"
title: "Test Section"
steps:
- step_id: "step_1"
title: "Question 1"
question: "What is 2+2?"
tokens_for_ai: "Categorize as 'correct' if answer is 4 or four, otherwise 'incorrect'"
buckets:
- correct
- incorrect
transitions:
correct:
content_blocks:
- "Great job!"
next_section_and_step: "section_1:step_2"
incorrect:
content_blocks:
- "Try again!"
counts_as_attempt: true
- step_id: "step_2"
title: "Question 2"
question: "What is 3+3?"
tokens_for_ai: "Categorize as 'correct' if answer is 6 or six, otherwise 'incorrect'"
buckets:
- correct
- incorrect
transitions:
correct:
content_blocks:
- "Excellent!"
incorrect:
content_blocks:
- "Not quite!"

25
research/tmplvez8rh9.yaml Normal file
View file

@ -0,0 +1,25 @@
default_max_attempts_per_step: 3
sections:
- section_id: "intro"
title: "Introduction"
steps:
- step_id: "info_1"
title: "Welcome"
content_blocks:
- "Welcome!"
- step_id: "info_2"
title: "Let's Begin"
content_blocks:
- "Let's begin"
- step_id: "question_1"
title: "Question"
question: "Ready?"
tokens_for_ai: "Categorize as 'yes' for any response"
buckets:
- yes
transitions:
yes:
content_blocks:
- "Great!"

37
research/tmpm0cwt68r.yaml Normal file
View file

@ -0,0 +1,37 @@
default_max_attempts_per_step: 3
sections:
- section_id: "section_1"
title: "Test Section"
steps:
- step_id: "step_1"
title: "Question 1"
question: "What is 2+2?"
tokens_for_ai: "Categorize as 'correct' if answer is 4 or four, otherwise 'incorrect'"
buckets:
- correct
- incorrect
transitions:
correct:
content_blocks:
- "Great job!"
next_section_and_step: "section_1:step_2"
incorrect:
content_blocks:
- "Try again!"
counts_as_attempt: true
- step_id: "step_2"
title: "Question 2"
question: "What is 3+3?"
tokens_for_ai: "Categorize as 'correct' if answer is 6 or six, otherwise 'incorrect'"
buckets:
- correct
- incorrect
transitions:
correct:
content_blocks:
- "Excellent!"
incorrect:
content_blocks:
- "Not quite!"

37
research/tmpm1bkqh1u.yaml Normal file
View file

@ -0,0 +1,37 @@
default_max_attempts_per_step: 3
sections:
- section_id: "section_1"
title: "Test Section"
steps:
- step_id: "step_1"
title: "Question 1"
question: "What is 2+2?"
tokens_for_ai: "Categorize as 'correct' if answer is 4 or four, otherwise 'incorrect'"
buckets:
- correct
- incorrect
transitions:
correct:
content_blocks:
- "Great job!"
next_section_and_step: "section_1:step_2"
incorrect:
content_blocks:
- "Try again!"
counts_as_attempt: true
- step_id: "step_2"
title: "Question 2"
question: "What is 3+3?"
tokens_for_ai: "Categorize as 'correct' if answer is 6 or six, otherwise 'incorrect'"
buckets:
- correct
- incorrect
transitions:
correct:
content_blocks:
- "Excellent!"
incorrect:
content_blocks:
- "Not quite!"

37
research/tmpp7gern8q.yaml Normal file
View file

@ -0,0 +1,37 @@
default_max_attempts_per_step: 3
sections:
- section_id: "section_1"
title: "Test Section"
steps:
- step_id: "step_1"
title: "Question 1"
question: "What is 2+2?"
tokens_for_ai: "Categorize as 'correct' if answer is 4 or four, otherwise 'incorrect'"
buckets:
- correct
- incorrect
transitions:
correct:
content_blocks:
- "Great job!"
next_section_and_step: "section_1:step_2"
incorrect:
content_blocks:
- "Try again!"
counts_as_attempt: true
- step_id: "step_2"
title: "Question 2"
question: "What is 3+3?"
tokens_for_ai: "Categorize as 'correct' if answer is 6 or six, otherwise 'incorrect'"
buckets:
- correct
- incorrect
transitions:
correct:
content_blocks:
- "Excellent!"
incorrect:
content_blocks:
- "Not quite!"

37
research/tmpsia1pfgk.yaml Normal file
View file

@ -0,0 +1,37 @@
default_max_attempts_per_step: 3
sections:
- section_id: "section_1"
title: "Test Section"
steps:
- step_id: "step_1"
title: "Question 1"
question: "What is 2+2?"
tokens_for_ai: "Categorize as 'correct' if answer is 4 or four, otherwise 'incorrect'"
buckets:
- correct
- incorrect
transitions:
correct:
content_blocks:
- "Great job!"
next_section_and_step: "section_1:step_2"
incorrect:
content_blocks:
- "Try again!"
counts_as_attempt: true
- step_id: "step_2"
title: "Question 2"
question: "What is 3+3?"
tokens_for_ai: "Categorize as 'correct' if answer is 6 or six, otherwise 'incorrect'"
buckets:
- correct
- incorrect
transitions:
correct:
content_blocks:
- "Excellent!"
incorrect:
content_blocks:
- "Not quite!"

37
research/tmpuo0kyov5.yaml Normal file
View file

@ -0,0 +1,37 @@
default_max_attempts_per_step: 3
sections:
- section_id: "section_1"
title: "Test Section"
steps:
- step_id: "step_1"
title: "Question 1"
question: "What is 2+2?"
tokens_for_ai: "Categorize as 'correct' if answer is 4 or four, otherwise 'incorrect'"
buckets:
- correct
- incorrect
transitions:
correct:
content_blocks:
- "Great job!"
next_section_and_step: "section_1:step_2"
incorrect:
content_blocks:
- "Try again!"
counts_as_attempt: true
- step_id: "step_2"
title: "Question 2"
question: "What is 3+3?"
tokens_for_ai: "Categorize as 'correct' if answer is 6 or six, otherwise 'incorrect'"
buckets:
- correct
- incorrect
transitions:
correct:
content_blocks:
- "Excellent!"
incorrect:
content_blocks:
- "Not quite!"

37
research/tmpus2gtcs3.yaml Normal file
View file

@ -0,0 +1,37 @@
default_max_attempts_per_step: 3
sections:
- section_id: "section_1"
title: "Test Section"
steps:
- step_id: "step_1"
title: "Question 1"
question: "What is 2+2?"
tokens_for_ai: "Categorize as 'correct' if answer is 4 or four, otherwise 'incorrect'"
buckets:
- correct
- incorrect
transitions:
correct:
content_blocks:
- "Great job!"
next_section_and_step: "section_1:step_2"
incorrect:
content_blocks:
- "Try again!"
counts_as_attempt: true
- step_id: "step_2"
title: "Question 2"
question: "What is 3+3?"
tokens_for_ai: "Categorize as 'correct' if answer is 6 or six, otherwise 'incorrect'"
buckets:
- correct
- incorrect
transitions:
correct:
content_blocks:
- "Excellent!"
incorrect:
content_blocks:
- "Not quite!"

25
research/tmpvplpyb_h.yaml Normal file
View file

@ -0,0 +1,25 @@
default_max_attempts_per_step: 3
sections:
- section_id: "intro"
title: "Introduction"
steps:
- step_id: "info_1"
title: "Welcome"
content_blocks:
- "Welcome!"
- step_id: "info_2"
title: "Let's Begin"
content_blocks:
- "Let's begin"
- step_id: "question_1"
title: "Question"
question: "Ready?"
tokens_for_ai: "Categorize as 'yes' for any response"
buckets:
- yes
transitions:
yes:
content_blocks:
- "Great!"

37
research/tmpx9mbpwnp.yaml Normal file
View file

@ -0,0 +1,37 @@
default_max_attempts_per_step: 3
sections:
- section_id: "section_1"
title: "Test Section"
steps:
- step_id: "step_1"
title: "Question 1"
question: "What is 2+2?"
tokens_for_ai: "Categorize as 'correct' if answer is 4 or four, otherwise 'incorrect'"
buckets:
- correct
- incorrect
transitions:
correct:
content_blocks:
- "Great job!"
next_section_and_step: "section_1:step_2"
incorrect:
content_blocks:
- "Try again!"
counts_as_attempt: true
- step_id: "step_2"
title: "Question 2"
question: "What is 3+3?"
tokens_for_ai: "Categorize as 'correct' if answer is 6 or six, otherwise 'incorrect'"
buckets:
- correct
- incorrect
transitions:
correct:
content_blocks:
- "Excellent!"
incorrect:
content_blocks:
- "Not quite!"

22
research/tmpzaep88gw.yaml Normal file
View file

@ -0,0 +1,22 @@
default_max_attempts_per_step: 3
sections:
- section_id: "section_1"
title: "Test Section"
steps:
- step_id: "step_1"
title: "Question 1"
question: "What is 2+2?"
tokens_for_ai: "Categorize as 'correct' if answer is 4 or four, otherwise 'incorrect'"
buckets:
- correct
- incorrect
transitions:
correct:
content_blocks:
- "Great job! Activity complete."
incorrect:
content_blocks:
- "Try again!"
counts_as_attempt: true

2
tests/functional/test_guarded_ai.py
View file

@ -434,7 +434,7 @@ class TestGuardedAIClientAndErrorHandling(unittest.TestCase):
"MODEL_API_KEY_2": "test-key-2",
}
with patch.dict(os.environ, test_env):
with patch.dict(os.environ, test_env, clear=False):
with patch("guarded_ai.get_client_for_endpoint") as mock_get_client:
mock_client1 = MagicMock()
mock_client2 = MagicMock()