May be MySQL creates a temporary table for intermediate results of the query, and that temporary table would grow too big to be kept in memory. Ensure with SHOW CREATE TABLE that the table actually is in InnoDB format.
-- This message may have been cut off and the rest will only be shown to members. To become a member, click here --