It’s been really fucking busy lately.
This is the end of my third week officially as a data engineer and it’s been consistently been getting more and more interesting and challenging every day.
The day to day activities are unimportant right now - what really has captured my attention has been the overlap between super nerdy topics like database design and equity. Beyond who is and who is not in your database, how well you design your database - when you work in the social tech sector - is really important.
Without going much into detail, imagine you work for a place that handles sensitive data and you provide crucial services, or connect people to such services, every day. Of course, a major negative event like your website going down or losing data will be a major L you take BUT.. what I’m thinking about is opportunity cost.
What if you are cruising along, stable growth, you’re helping people, every quarter you help more - you’re not going through the roof but you’re consistently growing healthily which means more and more people have access to those crucial services and everyone is happy.
What if, you could be doing 10% more by optimizing some part of your workflow (as a company, not even necessarily on your code)? 10% isn’t too much but these are crucial services so yeah, why not?
What if it was only 5%? 1%? What if it was -3% but it meant you’re moving to modern practices which eventually might lead to lower costs, more growth and what not?
I’m having trouble myself finding what is the best strategy here because in my opinion one person that could have had access to the services but didn’t because we didn’t try harder is one too many.
This example is more straight-forward than what I’ve been thinking, actually. What’s been on my mind lately is this scenario: Your company helps people, you have data on what you’ve been doing but your data is not in great shape to be analyzed for some very important questions (in my opinion) like what are the demographics of the people your company is helping? how many people do you help every week/month/quarter?
Like, imagine the data is there. Like, the data exists. Just, the database wasn’t designed for that so it takes you days or weeks to find an answer that kind of resembles what you’re being asked.
Is that impacting equity?
If it takes you two weeks to answer this type of questions, in my mind, you can only answer 26 of these questions a year. Each question answered leads you to make decisions that impact how you reach/impact people. If more information means better decision-making, which means more positive impact, then having a well-designed database so that you can answer these questions more quickly (and therefore allowing you to answer more questions overall) is an equity issue, right?