Measure simple enqueue vs AddEvent

Key Idea

Instead of insertEvents.js calling AddEvent directly, it will call a simple insert function to enqueue into a class Queue (similar to test_ProcessCreate)

Capture the events quickly first, scale out can come later

Main Goal

For this internship, measure the difference between throughput compared to https://www.notion.so/jymcheong/AddEvent-Throughput-3b2e5ee9b96e4c9db206d552908be3c3, meaning AddEvent vs Enqueue Function

Clarifications

Inserting "escaped" JSON string. Line 6 of test_ProcessCreate has a escape(JSON.stringify(pc)). This function returns a string, we want to insert that string into a Queue class.

This escaping of string is important, otherwise there will be problems.

Your last statement was:

so from this u can see the amount of time that can be saved if the events have already been processed before insertion am i right?

When we enqueue escaped string of an incoming event, it is NOT processed at all yet. I will be re-writing several ODB functions to Dequeue & change the way we derive Sequence. But AddEvent will largely be similar.

So this task is not so much of "time saved", but rather "timing difference" between a complex function like AddEvent vs a simple INSERT INTO QUEUE EventString = ... where ... is the escaped JSON object.

YJ's Documentation

Aim

Measure and compare the time it takes to insert 10000 ProcessCreate events using a simple enqueue function vs AddEvent function.

Implementation Steps

Create a new class called Queue in ODB. Create a property called EventString in class Queue to store the escaped JSON object.

Modify test_ProcessCreate function to insert the event into class Queue instead of calling AddEvent function.


db.command('INSERT INTO Queue SET EventString = ?',  escape(JSON.stringify(pc)))

Use the test function that was used in https://www.notion.so/jymcheong/AddEvent-Throughput-3b2e5ee9b96e4c9db206d552908be3c3 to measure the insertion performance.


//shortcut to get ticks in milliseconds since epoch
var s = (new Date())*1

print('start: ' + s)
print(Date())

for(var i=0;i < 10000;i++)
  test_ProcessCreate()

var e = (new Date())*1

print('end: ' + e)
print(Date())

var d = (e - s)/1000

print('====')
print('delta in secs ' + d)
print('====')

Results

As shown in the screenshot above, it took 6.3 seconds to finish inserting 10000 ProcessCreate events using the enqueue function.

Comparison

AddEvent function: 96 seconds to insert 10000 ProcessCreate events

Enqueue function: 6.3 seconds to insert 10000 ProcessCreate events

Enqueue function took 89.7 seconds less to finish inserting 10000 ProcessCreate events. That is almost 16 times faster than AddEvent function.