GoCB is Couchbase’s official SDK in golang. Since StoreDock’s Warehouse is Couchbase, We are able to get awesome sub-second queries to the database. This is solely done from Javascript. But however recently we faced a bottleneck.
Javascript was slower in performing a Task (In this case it was creating a snapshot of a certain doc we use called Daily Aggregation Doc). As the name suggests it runs daily and this is where javascript took over 30mins to process 130,000 such docs. The entire process involved:
-
SQS Push of 130K Objects
-
SQS Receive Msg of 130K messages
-
Couchbase 2 Reads and 1 Write per message (conditional write, but 99% of the time it was a write)
These 3 steps took over 30mins for a 6 workers in JavaScript consuming over 700-800MB (Since other services are also running)
Solution
Move to another home. So, we thought, If twitch.tv processes 10 billion messages per day on golang. It would be better to move such a repetitive operation to GoLang. Luckily couchbase has an official library called gocb.
But what’s the catch?
Well… Couchbase fails to deliver a stable SDK in my opinion. If we look at the godoc. Couchbase’s simple Get operation is returning a Golang error which is nothing but a string. There is literally no error code or error id 🙁
This makes it very hard as we have to rely on that string.
GO SDK:
bucket.Get(”DocumentID”)
returns “Key not found.” as error
NodeJS SDK:
bucket.Get(”DocumentID”)
returns { code: 11, error: “Key not found.“ } as error
Pro tip: Always return Error Codes.
Wanna see how we return error codes? 😀 This is how a StoreDock Error looks like:
Apart from:
err.code
We have something called:
subCodes
Sub Codes provide an abstract view on the flow of error. In the above code it says that there was a conflict in inserting and the document already exists. Hence the entire operation was errored because of that.