Scripted REST API, Stream Writers and HTTP Status

#Scripted REST API , #HTTP Status , #Stream Writer , #REST API Response Stream

Dave Slusher

4 minute read

tl;dr

If you use Stream Writers in your Scripted REST API, explicitly set the HTTP status or it won’t work the way you expect.

Muuuuch longer version

This is based on a problem a coworker (and sometimes cohost) saw yesterday. He set up a Scripted REST API that returned about 200 records in a JSON object, total payload was ~36k. When loaded in the browser directly, it worked fine. Consumed via either curl or a different instance with a REST message, nothing ever moved down the wire. By backing it down, he found that 5 records was the breaking point. would return in every case. If he tried 6 records, the earlier behavior was observed. Note that this is ~1K payload. Put a pin in that fact.

We spent some time working with this API trying to understand the boundaries of the problem. We looked at the code to see what was happening. The REST API was getting a copy of the Stream Writer via response.getStreamWriter() and used writeString() on the JSON.stringify() version of the JSON object. I had him explicitly set the Content-Type to application/json since his API only returned that. I also tried having him explicitly set the Content-Length based on the length of the string. Neither of these changes affected the behavior. What finally made it work was explicitly setting the HTTP status to 200.

Here are the headers I saw from curl initially (note the crucial first line):

< HTTP/1.1 0
< Set-Cookie: JSESSIONID=311E8B526A55E55E4BE83A5D5F6FDE75;Secure; Path=/; HttpOnly
< Set-Cookie: glide_user="";Secure; Expires=Thu, 01-Jan-1970 00:00:10 GMT; Path=/; HttpOnly
< Set-Cookie: glide\_user\_session="";Secure; Expires=Thu, 01-Jan-1970 00:00:10 GMT; Path=/; HttpOnly
< Set-Cookie: glide\_user\_route=glide.bef31b49336ea534f29ae78cac64cdb5;Secure; Expires=Tue, 17-Apr-2085 22:04:08 GMT; Path=/; HttpOnly
< X-Is-Logged-In: true
< Date: Thu, 30 Mar 2017 18:50:01 GMT
< Server: ServiceNow
< Set-Cookie: BIGipServerpool_cloudlabsdev=1518404618.52798.0000; path=/
< Strict-Transport-Security: max-age=15768000; includeSubDomains;
\* no chunk, no close, no size. Assume close to signal end

after explicitly setting status:

< HTTP/1.1 200 OK
< Set-Cookie: JSESSIONID=0B8578D91F69D93B276421FA2F69A053;Secure; Path=/; HttpOnly
< Set-Cookie: glide_user="";Secure; Expires=Thu, 01-Jan-1970 00:00:10 GMT; Path=/; HttpOnly
< Set-Cookie: glide\_user\_session="";Secure; Expires=Thu, 01-Jan-1970 00:00:10 GMT; Path=/; HttpOnly
< Set-Cookie: glide\_user\_route=glide.5fa39bd19498437ed84122e63fe58b3a;Secure; Expires=Tue, 17-Apr-2085 22:19:59 GMT; Path=/; HttpOnly
< X-Is-Logged-In: true
< Content-Type: application/json
< Transfer-Encoding: chunked
< Date: Thu, 30 Mar 2017 19:05:52 GMT
< Server: ServiceNow
< Set-Cookie: BIGipServerpool_cloudlabsdev=2659189770.57150.0000; path=/
< Strict-Transport-Security: max-age=15768000; includeSubDomains;

{“class_name”:“Some Lab3”, … [followed by the full 36K of the payload]

I posited the hypothesis (which was later confirmed) that this was related to the missing Content-Length or Transfer-Encoding headers. The reason the small payload works for the non-browser clients is because it all fits within a single 1k chunk. At the point the second chunk is received, curl or the REST Message don’t know what to do with it. They don’t know how much data to expect because there is no Content-Length and they haven’t been told to expect chunking. The browser is more forgiving because browsers have to be - it’s the same reason they don’t refuse to render malformed HTML. If they did, many seemingly ordinary transactions would fail.

There was some back and forth internally over whether this situation is as expected or represents a bug. The final ruling is this, which is the take-home lesson of the whole situation: when you get access to the Stream Writer, you not only have full control of the stream that is returned via the HTTP transaction but you must control that fully. This means also explicitly setting the Content-Type (which has a convenience API of .setContentType(String) ) and the HTTP status (via .setStatus(Number)). The rationale for this decision to not default the HTTP Status code to 200 if unset is that once you have taken control of the stream and the marshaling, it is up to you to determine success as the server can no longer infer that for you. If you don’t set 200 after you have requested the Stream Writer, it will not be inferred for you nor will the necessary headers be sent. In our case, the crucial one was “Transfer-Encoding: chunked” which was only set after the server was able to determine this was successful.

Final lesson

For anyone implementing Scripted REST APIs via controlling the output with Stream Writers, set all the values of the transaction or you will not get the behavior you expect.

Comments