Integer Overflow and operator::new
As Raymond Chen pointed out last year (https://blogs.msdn.com/oldnewthing/archive/2004/01/29/64389.aspx), there is a potential integer overflow when calling operator::new. The C++ compiler in Visual Studio 2005 automatically generates defensive code to mitigate this potential vulnerability.
Code like this:
class CFoo {
public:
CFoo() {m_p = NULL;}
private:
void *m_p;
};
void *func(size_t count) {
if (!count) return NULL;
return new CFoo[count];
}
When compiled with VS2005, yields output like this:
; _count$ = esi
; 14 : if (!count) return NULL;
00000 85 f6 test esi, esi
00002 74 36 je SHORT $LN4@func
; 15 : return new CFoo[count];
00004 33 c9 xor ecx, ecx
00006 8b c6 mov eax, esi
00008 ba 04 00 00 00 mov edx, 4
0000d f7 e2 mul edx
0000f 0f 90 c1 seto cl
00012 f7 d9 neg ecx
00014 0b c8 or ecx, eax
00016 51 push ecx
00017 e8 00 00 00 00 call ??2@YAPAXI@Z ; operator new
As you can see (assuming you can read assembly code!) the multiply occurs (mul edx) and then the CL register is set or not set depending on the value if the overflow flag. Now here’s the funky part, ECX will end up being 0x00000000 or 0xFFFFFFFF, and because of the next operation (or ecx) the ECX register will either be 0xFFFFFFFF or the value held in EAX, which is result of the initial multiply. This is then passed to operator::new which will fail in the face of 2^N-1 by returning NULL or throwing an exception.
It’s better to fail, and potentially crash the application, than have an integer overflow that may lead to code execution.
Of course, the correct solution is get the code right in the first place by performing good arithmetic hygiene, and in this case, restrict the maximum number of objects you’ll create:
void *func(size_t count) {
if (!count || count > MAX_FOO_ALLOWED) return NULL;
return new CFoo[count];
}
- Anonymous
December 06, 2005
The comment has been removed - Anonymous
December 12, 2005
So why not put an overflow check inside the implementation of new??
I would argue that if new CFoo[(1<<31)+1] runs a pointer to an array of [1] allocated object, that's a bug in new, and should be fixed in the RTL. - Anonymous
December 27, 2005
The mul instruction is the slowest way to multiply the number by 4. You can get the same result faster with LEA edx, [edx * 4] or with SHL edx, 2 or with ADD edx, edx / ADD edx, edx
What was optimization switches (/O1, /O2) when you compiled this code? - Anonymous
January 04, 2006
The comment has been removed - Anonymous
January 04, 2006
The comment has been removed - Anonymous
June 04, 2006
PingBack from http://codeka.com/blogs/index.php/dean/2006/06/05/integer_overflows_the_next_big_thing - Anonymous
January 10, 2007
MS07-004 does not affect Windows Vista, even though the coding bug is there. Why? The bug is an integer