Thta is an out-of-memory error returned by the serializer/deserializer. Try to split the output of the query into X json files.
Alternatively you can output to a varchar(max)
SELECT CAST((SELECT [Columns] FROM [Tables] FOR JSON .... AS VARCHAR(MAX)) AS JSONDATA
Or send the output to a temporary table also.
declare @json table (col1 nvarchar(max));
insert into @json
select ....