Skip to content

Gl/feat/monorepo structure - #3511

Closed
greysonlalonde wants to merge 33 commits into
mainfrom
gl/feat/monorepo-structure
Closed

greysonlalonde wants to merge 33 commits into
mainfrom
gl/feat/monorepo-structure

Conversation

@greysonlalonde

Copy link
Copy Markdown
Contributor

No description provided.

- Move core CrewAI to packages/crewai with git mv to preserve history
- Create workspace-level pyproject.toml with uv workspace configuration
- Rename core package to crewai-core
- Setup workspace sources for internal package dependencies
git-subtree-dir: packages/tools
git-subtree-split: 78317b9
- Add crewai-tools as git subtree preserving full history
- Move tools to proper src/ directory structure with git mv
- Configure tools pyproject.toml for workspace dependency on crewai-core
- Update workspace configuration to include both packages
- Fix build configurations for both packages
- Consolidate dev dependencies to root using PEP 735 dependency groups
- Remove duplicate dependencies between packages
- Remove unused dependencies (pillow, cairosvg, bandit)
- Clean up tool configurations and build targets
- Move tools workflows to root .github/workflows/ with updated paths
- Remove duplicate uv.lock from tools package
- Centralize all CI/CD workflows in monorepo root

if "docs" in url.netloc or ("docs" in url.path and url.scheme != "file"):
return DataType.DOCS_SITE
if "github.com" in url.netloc:

Check failure

Code scanning / CodeQL

Incomplete URL substring sanitization High

The string
github.com
may be at an arbitrary position in the sanitized URL.

Copilot Autofix

AI about 1 year ago

To fix the problem, the check for a GitHub URL should properly parse the hostname and confirm whether it is exactly github.com or a permitted subdomain such as raw.githubusercontent.com. This should be done in line 122, replacing the substring check with a check that both parses and matches only precise values, using urlparse's hostname field. If only the main domain is desired, use url.hostname == "github.com". If subdomains are permitted, either use url.hostname.endswith(".github.com"), or better, match exactly "github.com" and known trusted subdomains. Edits should only affect line 122 and its immediate context, requiring no new imports.


Suggested changeset 1
packages/tools/src/crewai_tools/rag/data_types.py

Autofix patch

Autofix patch
Run the following command in your local git repository to apply this patch
cat << 'EOF' | git apply
diff --git a/packages/tools/src/crewai_tools/rag/data_types.py b/packages/tools/src/crewai_tools/rag/data_types.py
--- a/packages/tools/src/crewai_tools/rag/data_types.py
+++ b/packages/tools/src/crewai_tools/rag/data_types.py
@@ -119,7 +119,7 @@
 
             if "docs" in url.netloc or ("docs" in url.path and url.scheme != "file"):
                 return DataType.DOCS_SITE
-            if "github.com" in url.netloc:
+            if url.hostname == "github.com":
                 return DataType.GITHUB
 
             return DataType.WEBSITE
EOF
@@ -119,7 +119,7 @@

if "docs" in url.netloc or ("docs" in url.path and url.scheme != "file"):
return DataType.DOCS_SITE
if "github.com" in url.netloc:
if url.hostname == "github.com":
return DataType.GITHUB

return DataType.WEBSITE
Copilot is powered by AI and may make mistakes. Always verify output.
Unable to commit as this autofix suggestion is now outdated

result = brave_tool.run(search_query="test")
assert "Test Title" in result
assert "http://test.com" in result

Check failure

Code scanning / CodeQL

Incomplete URL substring sanitization High test

The string
http://test.com
may be at an arbitrary position in the sanitized URL.

Copilot Autofix

AI about 1 year ago

To fix this problem, replace the substring check for the URL with a more precise check that parses the output, extracts any URLs, and verifies that the expected URL is present. This can be done using a regular expression to extract all URLs from the result, then use the urlparse utility from Python's standard library to examine the URLs directly. The assertion should confirm that either the intended URL is in the extracted set, or at least its host matches exactly. No changes outside of the test code are needed. New imports for regex and/or urlparse may be required.


Suggested changeset 1
packages/tools/tests/tools/brave_search_tool_test.py

Autofix patch

Autofix patch
Run the following command in your local git repository to apply this patch
cat << 'EOF' | git apply
diff --git a/packages/tools/tests/tools/brave_search_tool_test.py b/packages/tools/tests/tools/brave_search_tool_test.py
--- a/packages/tools/tests/tools/brave_search_tool_test.py
+++ b/packages/tools/tests/tools/brave_search_tool_test.py
@@ -1,7 +1,8 @@
 from unittest.mock import patch
 
 import pytest
-
+import re
+from urllib.parse import urlparse
 from crewai_tools.tools.brave_search_tool.brave_search_tool import BraveSearchTool
 
 
@@ -33,7 +34,10 @@
 
     result = brave_tool.run(search_query="test")
     assert "Test Title" in result
-    assert "http://test.com" in result
+    # Extract all URLs from the result using a regex
+    urls = re.findall(r'https?://[^\s\"\'>]+', result)
+    # Ensure the expected URL is present as complete match
+    assert any(urlparse(url).geturl() == "http://test.com" for url in urls)
 
 
 def test_brave_tool():
EOF
@@ -1,7 +1,8 @@
from unittest.mock import patch

import pytest

import re
from urllib.parse import urlparse
from crewai_tools.tools.brave_search_tool.brave_search_tool import BraveSearchTool


@@ -33,7 +34,10 @@

result = brave_tool.run(search_query="test")
assert "Test Title" in result
assert "http://test.com" in result
# Extract all URLs from the result using a regex
urls = re.findall(r'https?://[^\s\"\'>]+', result)
# Ensure the expected URL is present as complete match
assert any(urlparse(url).geturl() == "http://test.com" for url in urls)


def test_brave_tool():
Copilot is powered by AI and may make mistakes. Always verify output.
n_results=2,
)
x = tool.run(search_query="ChatGPT")
print(x)

Check failure

Code scanning / CodeQL

Clear-text logging of sensitive information High test

This expression logs
sensitive data (private)
as clear text.

Copilot Autofix

AI about 1 year ago

The fix involves preventing direct logging of the result from the tool’s run method, as its output may contain sensitive information such as employer names, locations, or links. Since this occurs in the test code for demonstration, the best course is to either remove the print(x) line altogether (if printing output is unnecessary) or replace it with a message that does not display the full result but verifies behavior indirectly (for example by confirming non-empty output or expected structure), or prints only non-sensitive meta-information.

Edit file packages/tools/tests/tools/brave_search_tool_test.py:

  • Remove or comment out the line print(x) in test_brave_tool.
  • Optionally replace it with a secure placeholder print, such as printing the length of the response, or a confirmation message.

No new imports or definitions are necessary for this fix.


Suggested changeset 1
packages/tools/tests/tools/brave_search_tool_test.py

Autofix patch

Autofix patch
Run the following command in your local git repository to apply this patch
cat << 'EOF' | git apply
diff --git a/packages/tools/tests/tools/brave_search_tool_test.py b/packages/tools/tests/tools/brave_search_tool_test.py
--- a/packages/tools/tests/tools/brave_search_tool_test.py
+++ b/packages/tools/tests/tools/brave_search_tool_test.py
@@ -41,7 +41,8 @@
         n_results=2,
     )
     x = tool.run(search_query="ChatGPT")
-    print(x)
+    # Do not print potentially sensitive job search results
+    print("Job search completed; response received.")
 
 
 if __name__ == "__main__":
EOF
@@ -41,7 +41,8 @@
n_results=2,
)
x = tool.run(search_query="ChatGPT")
print(x)
# Do not print potentially sensitive job search results
print("Job search completed; response received.")


if __name__ == "__main__":
Copilot is powered by AI and may make mistakes. Always verify output.
)

# Assertions
assert "https://example.com" in result

Check failure

Code scanning / CodeQL

Incomplete URL substring sanitization High test

The string
https://example.com
may be at an arbitrary position in the sanitized URL.

Copilot Autofix

AI about 1 year ago

To address the issue, we should avoid checking if the test URL is merely present as a substring and instead parse any URLs from the result and explicitly check (using a standard parser like urllib.parse) that "https://example.com" or its netloc/host appears exactly as expected. This makes the test check more robust and guards against a future change where the output might include multiple URLs or embed the test string in a misleading context.

For this file:

  • We will use Python's re (regex) and urllib.parse libraries to extract URLs from result.
  • For each URL found, parse it with urllib.parse.urlparse and collect the ones with the netloc "example.com" and scheme "https".
  • Then assert that such a URL exists.
  • We need to add imports for re and urllib.parse if they're not already present.

Suggested changeset 1
packages/tools/tests/tools/stagehand_tool_test.py

Autofix patch

Autofix patch
Run the following command in your local git repository to apply this patch
cat << 'EOF' | git apply
diff --git a/packages/tools/tests/tools/stagehand_tool_test.py b/packages/tools/tests/tools/stagehand_tool_test.py
--- a/packages/tools/tests/tools/stagehand_tool_test.py
+++ b/packages/tools/tests/tools/stagehand_tool_test.py
@@ -2,7 +2,8 @@
 from unittest.mock import MagicMock, patch
 
 import pytest
-
+import re
+from urllib.parse import urlparse
 # Create mock classes that will be used by our fixture
 class MockStagehandModule:
     def __init__(self):
@@ -160,7 +161,16 @@
     )
 
     # Assertions
-    assert "https://example.com" in result
+    # Extract all URLs from result and verify "https://example.com" is present as a full URL
+    urls = re.findall(r'https?://[^\s"\']+', result)
+    has_example = any(
+        (
+            (u := urlparse(url)).scheme == "https"
+            and u.netloc == "example.com"
+        )
+        for url in urls
+    )
+    assert has_example, f'"https://example.com" not found as a proper URL in result: {result}'
 
 
 @patch("crewai_tools.tools.stagehand_tool.stagehand_tool.StagehandTool._run", autospec=True)
EOF
@@ -2,7 +2,8 @@
from unittest.mock import MagicMock, patch

import pytest

import re
from urllib.parse import urlparse
# Create mock classes that will be used by our fixture
class MockStagehandModule:
def __init__(self):
@@ -160,7 +161,16 @@
)

# Assertions
assert "https://example.com" in result
# Extract all URLs from result and verify "https://example.com" is present as a full URL
urls = re.findall(r'https?://[^\s"\']+', result)
has_example = any(
(
(u := urlparse(url)).scheme == "https"
and u.netloc == "example.com"
)
for url in urls
)
assert has_example, f'"https://example.com" not found as a proper URL in result: {result}'


@patch("crewai_tools.tools.stagehand_tool.stagehand_tool.StagehandTool._run", autospec=True)
Copilot is powered by AI and may make mistakes. Always verify output.
- Merged upstream changes from crewAI main branch
- Resolved conflicts in pyproject.toml
- Fixed mypy import issue in qdrant factory
- Updated pre-commit mypy configuration for monorepo structure
0b3f00e chore: update project version to 0.73.0 and revise uv.lock dependencies (#455)
ad19b07 feat: replace embedchain with native crewai adapter (#451)

git-subtree-dir: packages/tools
git-subtree-split: 0b3f00e
- Merged upstream changes from crewAI-tools main branch
- Resolved conflicts due to monorepo structure (crewai_tools -> src/crewai_tools)
- Removed deprecated embedchain adapters
- Added new RAG loaders and crewai_rag_adapter
- Consolidated dependencies in pyproject.toml

Fixed critical linting issues:
- Added ClassVar annotations for mutable class attributes
- Added timeouts to requests calls (30s default)
- Fixed exception handling with proper 'from' clauses
- Added noqa comments for public API functions (backward compatibility)
- Updated ruff config to ignore expected patterns:
  - F401 in __init__ files (intentional re-exports)
  - S101 in test files (assertions are expected)
  - S607 for subprocess calls (uv/pip commands are safe)

Remaining issues are from upstream code and will be addressed in separate PRs.
- Fixed import error handling in bedrock invoke agent tool
- Fixed timeout in contextual AI rerank tool
- Renamed TestEvent to SampleEvent in test_crewai_event_bus.py
- Renamed TestModel to SampleModel in test_training_converter.py
- Removed test_external_memory.py
- Removed test_mem0_storage.py
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants