Repository navigation
Gl/feat/monorepo structure - #3511
greysonlalonde wants to merge 33 commits into
Conversation
- Move core CrewAI to packages/crewai with git mv to preserve history - Create workspace-level pyproject.toml with uv workspace configuration - Rename core package to crewai-core - Setup workspace sources for internal package dependencies
git-subtree-dir: packages/tools git-subtree-split: 78317b9
- Add crewai-tools as git subtree preserving full history - Move tools to proper src/ directory structure with git mv - Configure tools pyproject.toml for workspace dependency on crewai-core - Update workspace configuration to include both packages - Fix build configurations for both packages
- Consolidate dev dependencies to root using PEP 735 dependency groups - Remove duplicate dependencies between packages - Remove unused dependencies (pillow, cairosvg, bandit) - Clean up tool configurations and build targets
- Move tools workflows to root .github/workflows/ with updated paths - Remove duplicate uv.lock from tools package - Centralize all CI/CD workflows in monorepo root
|
|
||
| if "docs" in url.netloc or ("docs" in url.path and url.scheme != "file"): | ||
| return DataType.DOCS_SITE | ||
| if "github.com" in url.netloc: |
Check failure
Code scanning / CodeQL
Incomplete URL substring sanitization High
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI about 1 year ago
To fix the problem, the check for a GitHub URL should properly parse the hostname and confirm whether it is exactly github.com or a permitted subdomain such as raw.githubusercontent.com. This should be done in line 122, replacing the substring check with a check that both parses and matches only precise values, using urlparse's hostname field. If only the main domain is desired, use url.hostname == "github.com". If subdomains are permitted, either use url.hostname.endswith(".github.com"), or better, match exactly "github.com" and known trusted subdomains. Edits should only affect line 122 and its immediate context, requiring no new imports.
| @@ -119,7 +119,7 @@ | ||
|
|
||
| if "docs" in url.netloc or ("docs" in url.path and url.scheme != "file"): | ||
| return DataType.DOCS_SITE | ||
| if "github.com" in url.netloc: | ||
| if url.hostname == "github.com": | ||
| return DataType.GITHUB | ||
|
|
||
| return DataType.WEBSITE |
|
|
||
| result = brave_tool.run(search_query="test") | ||
| assert "Test Title" in result | ||
| assert "http://test.com" in result |
Check failure
Code scanning / CodeQL
Incomplete URL substring sanitization High test
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI about 1 year ago
To fix this problem, replace the substring check for the URL with a more precise check that parses the output, extracts any URLs, and verifies that the expected URL is present. This can be done using a regular expression to extract all URLs from the result, then use the urlparse utility from Python's standard library to examine the URLs directly. The assertion should confirm that either the intended URL is in the extracted set, or at least its host matches exactly. No changes outside of the test code are needed. New imports for regex and/or urlparse may be required.
| @@ -1,7 +1,8 @@ | ||
| from unittest.mock import patch | ||
|
|
||
| import pytest | ||
|
|
||
| import re | ||
| from urllib.parse import urlparse | ||
| from crewai_tools.tools.brave_search_tool.brave_search_tool import BraveSearchTool | ||
|
|
||
|
|
||
| @@ -33,7 +34,10 @@ | ||
|
|
||
| result = brave_tool.run(search_query="test") | ||
| assert "Test Title" in result | ||
| assert "http://test.com" in result | ||
| # Extract all URLs from the result using a regex | ||
| urls = re.findall(r'https?://[^\s\"\'>]+', result) | ||
| # Ensure the expected URL is present as complete match | ||
| assert any(urlparse(url).geturl() == "http://test.com" for url in urls) | ||
|
|
||
|
|
||
| def test_brave_tool(): |
| n_results=2, | ||
| ) | ||
| x = tool.run(search_query="ChatGPT") | ||
| print(x) |
Check failure
Code scanning / CodeQL
Clear-text logging of sensitive information High test
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI about 1 year ago
The fix involves preventing direct logging of the result from the tool’s run method, as its output may contain sensitive information such as employer names, locations, or links. Since this occurs in the test code for demonstration, the best course is to either remove the print(x) line altogether (if printing output is unnecessary) or replace it with a message that does not display the full result but verifies behavior indirectly (for example by confirming non-empty output or expected structure), or prints only non-sensitive meta-information.
Edit file packages/tools/tests/tools/brave_search_tool_test.py:
- Remove or comment out the line
print(x)intest_brave_tool. - Optionally replace it with a secure placeholder print, such as printing the length of the response, or a confirmation message.
No new imports or definitions are necessary for this fix.
| @@ -41,7 +41,8 @@ | ||
| n_results=2, | ||
| ) | ||
| x = tool.run(search_query="ChatGPT") | ||
| print(x) | ||
| # Do not print potentially sensitive job search results | ||
| print("Job search completed; response received.") | ||
|
|
||
|
|
||
| if __name__ == "__main__": |
| ) | ||
|
|
||
| # Assertions | ||
| assert "https://example.com" in result |
Check failure
Code scanning / CodeQL
Incomplete URL substring sanitization High test
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI about 1 year ago
To address the issue, we should avoid checking if the test URL is merely present as a substring and instead parse any URLs from the result and explicitly check (using a standard parser like urllib.parse) that "https://example.com" or its netloc/host appears exactly as expected. This makes the test check more robust and guards against a future change where the output might include multiple URLs or embed the test string in a misleading context.
For this file:
- We will use Python's
re(regex) andurllib.parselibraries to extract URLs fromresult. - For each URL found, parse it with
urllib.parse.urlparseand collect the ones with the netloc "example.com" and scheme "https". - Then assert that such a URL exists.
- We need to add imports for
reandurllib.parseif they're not already present.
| @@ -2,7 +2,8 @@ | ||
| from unittest.mock import MagicMock, patch | ||
|
|
||
| import pytest | ||
|
|
||
| import re | ||
| from urllib.parse import urlparse | ||
| # Create mock classes that will be used by our fixture | ||
| class MockStagehandModule: | ||
| def __init__(self): | ||
| @@ -160,7 +161,16 @@ | ||
| ) | ||
|
|
||
| # Assertions | ||
| assert "https://example.com" in result | ||
| # Extract all URLs from result and verify "https://example.com" is present as a full URL | ||
| urls = re.findall(r'https?://[^\s"\']+', result) | ||
| has_example = any( | ||
| ( | ||
| (u := urlparse(url)).scheme == "https" | ||
| and u.netloc == "example.com" | ||
| ) | ||
| for url in urls | ||
| ) | ||
| assert has_example, f'"https://example.com" not found as a proper URL in result: {result}' | ||
|
|
||
|
|
||
| @patch("crewai_tools.tools.stagehand_tool.stagehand_tool.StagehandTool._run", autospec=True) |
- Merged upstream changes from crewAI main branch - Resolved conflicts in pyproject.toml - Fixed mypy import issue in qdrant factory - Updated pre-commit mypy configuration for monorepo structure
- Merged upstream changes from crewAI-tools main branch - Resolved conflicts due to monorepo structure (crewai_tools -> src/crewai_tools) - Removed deprecated embedchain adapters - Added new RAG loaders and crewai_rag_adapter - Consolidated dependencies in pyproject.toml Fixed critical linting issues: - Added ClassVar annotations for mutable class attributes - Added timeouts to requests calls (30s default) - Fixed exception handling with proper 'from' clauses - Added noqa comments for public API functions (backward compatibility) - Updated ruff config to ignore expected patterns: - F401 in __init__ files (intentional re-exports) - S101 in test files (assertions are expected) - S607 for subprocess calls (uv/pip commands are safe) Remaining issues are from upstream code and will be addressed in separate PRs.
- Fixed import error handling in bedrock invoke agent tool - Fixed timeout in contextual AI rerank tool
- Renamed TestEvent to SampleEvent in test_crewai_event_bus.py - Renamed TestModel to SampleModel in test_training_converter.py
- Removed test_external_memory.py - Removed test_mem0_storage.py
No description provided.