Repository navigation
fix: generate signed byte types for VT_I1 - #962
haoxuanjng-lang wants to merge 3 commits into
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #962 +/- ##
==========================================
+ Coverage 89.21% 89.23% +0.02%
==========================================
Files 142 143 +1
Lines 13870 13924 +54
==========================================
+ Hits 12374 12425 +51
- Misses 1496 1499 +3 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
junkmd
left a comment
There was a problem hiding this comment.
Thank you for your continued contributions!
This is clearly an inconsistency, and it is normally not the correct behavior for software. However, I could not find any earlier bug reports about this specific behavior. This makes me wonder whether users have come to expect the current behavior over time, or whether no one has ever reached this code path.
So, let's move forward step by step, checking both the impact of this change and backward compatibility.
As before, let's start with the test structure.
junkmd
left a comment
There was a problem hiding this comment.
Thank you for adding the native COM test.
It shows that signed 8-bit integers work correctly across the COM boundary in Echo....
It also confirms that explicit LPSTR still uses the existing string handling.
These tests are useful for checking number marshaling and making sure existing string handling still works. However, I noticed that these are distinct concerns.
We also need to check whether existing Python calls remain compatible with or break the new mapping.
Checking Backward Compatibility
To better understand the change, I wrote a parameterized test that passes different types of 1-byte data as arguments. I compared the results before and after the change in tlbparser.
For this test, I expected the return value to stay the same, even when the argument type changed.
def test_backward_compatibility(self):
for i, value in enumerate(
[
b"h",
pointer(c_byte(ord("h"))),
byref(c_byte(ord("h"))),
(c_byte * 1)(ord("h")),
]
):
with self.subTest(index=i, value=value):
self.assertEqual(self.server.ReadCharPointer(value), ord("h"))The results differ before and after the change.
Before the change, b"h" works, but the other three arguments raise ctypes.ArgumentError.
test_backward_compatibility (comtypes.test.test_cpp_byte_echo.ByteEchoTest.test_backward_compatibility) ...
test_backward_compatibility (comtypes.test.test_cpp_byte_echo.ByteEchoTest.test_backward_compatibility) (index=1, value=<ctypes.LP_c_byte object at 0x0000025878B3F250>) ... ERROR
test_backward_compatibility (comtypes.test.test_cpp_byte_echo.ByteEchoTest.test_backward_compatibility) (index=2, value=<cparam 'P' (0x0000025878F938D8)>) ... ERROR
test_backward_compatibility (comtypes.test.test_cpp_byte_echo.ByteEchoTest.test_backward_compatibility) (index=3, value=<comtypes.test.test_cpp_byte_echo.c_byte_Array_1 object at 0x0000025878F93A20>) ... ERROR
======================================================================
ERROR: test_backward_compatibility (comtypes.test.test_cpp_byte_echo.ByteEchoTest.test_backward_compatibility) (index=1, value=<ctypes.LP_c_byte object at 0x0000025878B3F250>)
----------------------------------------------------------------------
Traceback (most recent call last):
File "path\to\comtypes\test\test_cpp_byte_echo.py", line 65, in test_backward_compatibility
self.assertEqual(self.server.ReadCharPointer(value), ord("h"))
~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^
ctypes.ArgumentError: argument 1: TypeError: 'LP_c_byte' object cannot be interpreted as ctypes.c_char_p
======================================================================
ERROR: test_backward_compatibility (comtypes.test.test_cpp_byte_echo.ByteEchoTest.test_backward_compatibility) (index=2, value=<cparam 'P' (0x0000025878F938D8)>)
----------------------------------------------------------------------
Traceback (most recent call last):
File "path\to\comtypes\test\test_cpp_byte_echo.py", line 65, in test_backward_compatibility
self.assertEqual(self.server.ReadCharPointer(value), ord("h"))
~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^
ctypes.ArgumentError: argument 1: TypeError: '_ctypes.CArgObject' object cannot be interpreted as ctypes.c_char_p
======================================================================
ERROR: test_backward_compatibility (comtypes.test.test_cpp_byte_echo.ByteEchoTest.test_backward_compatibility) (index=3, value=<comtypes.test.test_cpp_byte_echo.c_byte_Array_1 object at 0x0000025878F93A20>)
----------------------------------------------------------------------
Traceback (most recent call last):
File "path\to\comtypes\test\test_cpp_byte_echo.py", line 65, in test_backward_compatibility
self.assertEqual(self.server.ReadCharPointer(value), ord("h"))
~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^
ctypes.ArgumentError: argument 1: TypeError: 'c_byte_Array_1' object cannot be interpreted as ctypes.c_char_p
----------------------------------------------------------------------
Ran 1 test in 0.027s
FAILED (errors=3)Generated Code (Only `IByteEchoTest`)
class IByteEchoTest(comtypes.gen._00020430_0000_0000_C000_000000000046_0_2_0.IUnknown):
_case_insensitive_ = True
_iid_ = GUID('{047E34C1-33A8-4DAA-A1E5-B941756A8319}')
_idlflags_ = ['oleautomation']
if TYPE_CHECKING: # commembers
def EchoInt8(self, value: hints.Incomplete) -> hints.Incomplete: ...
def EchoUint8(self, value: hints.Incomplete) -> hints.Incomplete: ...
def EchoLpStr(self, value: hints.Incomplete) -> hints.Incomplete: ...
def ReadCharPointer(self, value: hints.Incomplete) -> hints.Incomplete: ...
IByteEchoTest._methods_ = [
COMMETHOD(
[],
HRESULT,
'EchoInt8',
(['in'], c_char, 'value'),
(['out', 'retval'], STRING, 'result')
),
COMMETHOD(
[],
HRESULT,
'EchoUint8',
(['in'], c_ubyte, 'value'),
(['out', 'retval'], POINTER(c_ubyte), 'result')
),
COMMETHOD(
[],
HRESULT,
'EchoLpStr',
(['in'], STRING, 'value'),
(['out', 'retval'], POINTER(STRING), 'result')
),
COMMETHOD(
[],
HRESULT,
'ReadCharPointer',
(['in'], STRING, 'value'),
(['out', 'retval'], POINTER(c_ubyte), 'result')
),
]↓
After the change, the other three arguments work, but b"h" raises ctypes.ArgumentError.
test_backward_compatibility (comtypes.test.test_cpp_byte_echo.ByteEchoTest.test_backward_compatibility) ...
test_backward_compatibility (comtypes.test.test_cpp_byte_echo.ByteEchoTest.test_backward_compatibility) (index=0, value=b'h') ... ERROR
======================================================================
ERROR: test_backward_compatibility (comtypes.test.test_cpp_byte_echo.ByteEchoTest.test_backward_compatibility) (index=0, value=b'h')
----------------------------------------------------------------------
Traceback (most recent call last):
File "path\to\comtypes\test\test_cpp_byte_echo.py", line 65, in test_backward_compatibility
self.assertEqual(self.server.ReadCharPointer(value), ord("h"))
~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^
ctypes.ArgumentError: argument 1: TypeError: expected LP_c_byte instance instead of bytes
----------------------------------------------------------------------
Ran 1 test in 0.025s
FAILED (errors=1)Generated Code (Only `IByteEchoTest`)
class IByteEchoTest(comtypes.gen._00020430_0000_0000_C000_000000000046_0_2_0.IUnknown):
_case_insensitive_ = True
_iid_ = GUID('{047E34C1-33A8-4DAA-A1E5-B941756A8319}')
_idlflags_ = ['oleautomation']
if TYPE_CHECKING: # commembers
def EchoInt8(self, value: hints.Incomplete) -> hints.Incomplete: ...
def EchoUint8(self, value: hints.Incomplete) -> hints.Incomplete: ...
def EchoLpStr(self, value: hints.Incomplete) -> hints.Incomplete: ...
def ReadCharPointer(self, value: hints.Incomplete) -> hints.Incomplete: ...
IByteEchoTest._methods_ = [
COMMETHOD(
[],
HRESULT,
'EchoInt8',
(['in'], c_byte, 'value'),
(['out', 'retval'], POINTER(c_byte), 'result')
),
COMMETHOD(
[],
HRESULT,
'EchoUint8',
(['in'], c_ubyte, 'value'),
(['out', 'retval'], POINTER(c_ubyte), 'result')
),
COMMETHOD(
[],
HRESULT,
'EchoLpStr',
(['in'], STRING, 'value'),
(['out', 'retval'], POINTER(STRING), 'result')
),
COMMETHOD(
[],
HRESULT,
'ReadCharPointer',
(['in'], POINTER(c_byte), 'value'),
(['out', 'retval'], POINTER(c_ubyte), 'result')
),
]The return value stays the same for the calls that succeed.
These results show that the new mapping allows calls that match the new type, but breaks some calls that worked with the old type. In other words, the accepted argument types change, even though the data still represents the same 1-byte value.
Checking the Other Direction
I think we should also test the opposite case of ReadCharPointer.
For example, we could add a method like GetCharacterBytes. It would take an integer such as ord("h") and return a Python bytes object such as b"h".
We should check what happens to existing calls after the mapping changes. Does the call raise ctypes.ArgumentError, or does it succeed but return a different value, causing the assertion to fail?
We should then test different combinations of argument and return types to find out which combinations pass. For this investigation, we can set aside whether each combination meets the original requirements.
By testing both directions, we can better understand how the mapping change affects arguments and return values, and what changes users may need to make to their Python code.
Docs for Compatibility
Some users define their interfaces in Python by inheriting from IUnknown or IDispatch. Others keep the generated module source code in their own repositories. These users may be able to fix problems by changing the type definitions themselves.
However, users who use modules generated from type libraries without changing them cannot easily fix the generated types. They need to know why the types have changed, which existing calls are affected, and how to update their code.
If we move forward with this change, I think we should explain its impact and how to migrate in a migration guide or the release notes.
In particular, users who use generated modules as they are need enough information to decide what changes to make. We should explain:
- Which
VT_I1arguments and return values will have different generated types. - Which existing calls will stop working.
- Which values or
ctypestypes users should pass after the change. - How explicitly declared string types, such as
LPSTR, are handled.
Alternative Approach
Another option is to choose the type based on more detailed information instead of looking it up in the COMTYPES mapping by VT_* alone. If we consider this approach, we may need a different test double to test it.
I think these tests will help us understand the compatibility impact before deciding how to proceed.
|
I am planning to add a component to the C++ test double to address another open issue in this project. This may cause conflicts with this PR, so please handle them if they occur. |
|
Added the reverse-direction probe and expanded the real native-call matrix. The separate component/interface/test module are now named The output probe writes one character through a plain Observed with freshly generated wrappers against latest-main production versus the proposed mapping:
This establishes the opposite direction's accepted-buffer change independently of SAFEARRAYs. Explicit LPSTR and unsigned scalar tables still pass both versions. I retained the two legacy input/output assertions as expected failures and added documentation for affected generated types, saved/handwritten interfaces, replacement arguments, signed outputs and explicit string handling. Full local unittest: 474 run / 448 passed / 24 skipped / 2 expected compatibility failures. Focused suites: 70 / 59 / 9 / 2; without NumPy/pywin32: 51 / 41 / 8 / 2. Native build, full Ruff CI scope and strict Sphinx HTML build pass. Existing full docs doctest cannot get past its optional Excel type-library setup locally ( The mapping itself is unchanged in this follow-up and the PR remains draft. These results and the documentation describe the compatibility cost; they do not resolve it or decide whether a more detailed mapping should be used. |
junkmd
left a comment
There was a problem hiding this comment.
Thank you for adding the new test double!
I am going to add another test double to source/CppTestSrv/ as well.
If any conflicts happen, it would be great if you could resolve them again.
| def test_character_pointer_output(self): | ||
| for value in (0, 1, ord("h"), 128, 255): | ||
| with self.subTest(value=value): | ||
| # Plain char* output uses the signed numeric mapping as well. | ||
| self.assertEqual( | ||
| self.server.GetCharacterBytes(value), | ||
| value if value < 128 else value - 256, | ||
| ) | ||
|
|
||
| def test_character_ctypes_output_buffers(self): | ||
| for value in (0, ord("h"), 128, 255): | ||
| for result in (c_byte(), (c_byte * 1)()): | ||
| with self.subTest(value=value, result=result): | ||
| argument = byref(result) if isinstance(result, c_byte) else result | ||
| self.server._IByteCompatibilityTest__com_GetCharacterBytes( | ||
| value, argument | ||
| ) | ||
| actual = result.value if isinstance(result, c_byte) else result[0] | ||
| self.assertEqual(actual, value if value < 128 else value - 256) |
There was a problem hiding this comment.
Having if statements in the test and using values that are not fixed makes it a bit hard for me to read.
If the way we check values changes depending on the data type in parametrize, it might be better to split them into separate tests.
If the type is the same but the values are different, adding the expected value to the test parameters would make it easier to compare.
It could also be helpful to separate the test cases for old-mapping vs proposed-mapping.
I'm having trouble picturing the full behavior here. Could you walk me through it with test codebases?
Generating
VT_I1asc_charturns the native fixture's-7into249in a tuple, and produces byte-string arrays or a reshape error with NumPy. This PR proposessigned char/c_byteforVT_I1; explicitVT_LPSTRremains a narrow string. Follow-up to #956 and the discussion in #935; this does not reproduce or resolve #935's intermittent UIAord()exception.Draft: native calls confirm backward-compatibility regressions in both input and output directions. These are documented, not fixed.
ReadCharPointer(b"h")works with the old mapping and raisesctypes.ArgumentErrorwith the new mapping. The generated rawGetCharacterBytescall similarly accepts a writable character buffer with the old mapping and rejects it with the new one. Both legacy assertions remain explicit expected failures pending the compatibility decision.The separate native component is now
CoByteCompatibilityTest/IByteCompatibilityTest. Its generated interface exercises signed/unsigned scalar echoes, explicit LPSTR, plain char-pointer input, and plain char-pointer output. The new output method writes exactly one character into caller-owned storage; it does not assume a native NUL terminator or allocate a returned string. The legacy output probe reserves its own trailing NUL in a two-byte caller buffer and obtainsb"h"from that buffer. Automatic high-level output allocation fails on the old mapping withc_char_p 'out' parameter must be passed as default value; the new mapping returns signed integers. Tests cover pointer/reference/array inputs, numeric outputs including 128 -> -128 and 255 -> -1, and numeric output buffers. Explicit LPSTR still covers empty, ASCII and high-bit bytes, retaining/freeing its allocated output through the generated raw method.Independent numeric SAFEARRAY 1-D/2-D tuple and NumPy value/shape/dtype/copy checks remain. No original published test-interface ABI is changed. Merged current main
939341cdf17e4c282b056bffc9e7bc70899ca756and resolved conflicts in the Makefile, factory table and IDL, preserving the newly upstreamed string-SAFEARRAY-in-VARIANT component. Added a compatibility/migration page explaining regenerated versus saved/handwritten interfaces, affected calls, replacement ctypes values, signed return values and unchanged explicit strings.Validation on Windows x64, Python 3.13.5, NumPy 2.5.3, pywin32 312, MSVC 19.42:
nmake /f MAKEFILE. Fresh generated fixture wrappers for each baseline/fixed stage; actual out-of-process COM calls using temporary HKCU registrations with cleanup verified.test_cpp_safearray,test_cpp_byte_compatibility,test_npsupport,test_safearray,test_variant): 70 run / 59 passed / 9 skipped / 2 expected compatibility failures.python -m unittest discover -v -s comtypes/test -t comtypes/test: 474 run / 448 passed / 24 skipped / 2 expected compatibility failures. Includes the newly merged native string-SAFEARRAY-in-VARIANT tests. Optional Office/server/admin coverage remains skipped.git diff --checkpasses.docs/requirements.txt: passes. The full existing docs doctest exits before examples because its global setup tries an unregistered Excel type library (WinError -2147319779); that environment limitation is not reported as a pass.CI for this revision is pending; the preceding head's 65 successes do not validate it. The production mapping proposal remains unchanged by this review follow-up. This stays draft until the compatibility strategy is agreed; the two expected failures do not mean compatibility is resolved.
AI assistance: OpenAI Codex helped investigate, implement, and validate this contribution.