💪
YJ's Internship/OpenEDR Sizing/
Storage Estimation
Storage Estimation

Storage Estimation

Assign
Date
‣
Status
Completed
Property

Scenario

  • Suppose a company has 100 Win10 endpoint
  • Every employee will boot up their machine at 8am & shutdown at 5pm
  • Each employee will run max of 5 apps daily for their job
  • Audit-&-Compliance requires all activities from these endpoints to be recorded for 3 months.

Goals & Approach

  1. Propose ways to estimate storage sizing for discussion
  1. After review & discussion, execute the test plan to achieve the estimates

Proposed Approach

  1. Wipe out all data in ODB. Get size of clean ODB with no data.
      2. var db = orient.getDatabase();
var size = db.getSize()
print(size) //in bytes
  1. Collect logs on a single Win VM for 30 minutes. During the 30 minutes, boot up the machine and run 5 applications i.e. I.E., Chrome, Firefox, Outlook & MS Word.
  1. Get size of ODB again. New size - old size is an estimate of how much data 1 Win10 endpoint will generate in 30 minutes. Multiply by 18 to get size of data generated in a day. (9hrs ⇒ 540mins / 30 mins = 18)^^
  1. Multiply by 100 to get estimated size of data from 100 Win10 endpoints in a day.
  1. 3 months → around 90 days. Multiply by 90 to get estimated size of storage needed to store data from 100 Win10 endpoints for 3 months.
  1. Answer from part 5 + size of clean ODB = total storage needed for ODB.

Executing Approach

Size of freshly installed ODB with NO events = 327,126 bytes
Size of ODB after collecting data for 30 mins = 5,243,459 bytes
Size of data generated from 30 mins (inclusive of machine bootup) = 5,243,459 - 327,126 = 4,916,333 bytes
Size of data generated during bootup of machine = 3,000,000 bytes
Size of data generated from 30 mins (exclusive of machine bootup) = 4,916,333 - 3,000,000^ = 1,916,333 bytes
❓
^3,000,000 is an estimate of the Windows endpoint boot-up-data size. We are not waiting for 9 hours to collect in view of time-constraint. Instead, we estimate the 30 mins worth of data WITHOUT boot-up-data size. After which, as you can see in the subsequent step to multiply by 18^^ to extrapolate to 9 hours worth of data. For a better estimate, one may want to let the endpoint(s) run for 9 hours (as per our scenario, or whatever timeframe suitable for your environment)
Size of data from 1 Win10 endpoint in a day (9hrs) = 3,000,000 + (1,916,333 x 18) = 35.7571 MB
Size of data from 100 Win10 endpoints in a day = 35.7571 MB x 100 = 3.4919 GB
Size of data from 100 Win10 endpoints for 3 months = 3.4919 GB x 90 = 314.271 GB
Estimated total storage needed = 314.271 GB + 327,126 bytes = 314.30 GB
 
Conversion calculator: https://convertlive.com/u/convert/bytes/to/megabytes

Guidance & Suggestions

Where are the ODB data stored?

notion image

Wiping out ODB & MongoDB

Look for these scripts within your backend installation; "backend" & "frontend" respectively