Programming

[kerowo]

Why is names in it’s own table instead of a column on Entity?

Put an index on all of the columns you are going to search by, so at least id and maybe type.

Without indexes my guess is every query is going a full table scan, which is slowing you down.

[bobman0330]

Thanks! I have a separate name table to deal with name changes over time.

[microbet]

There are multiple names per entry, right?

Primary key on the name table?

[bobman0330]

There are (or at least can be). PK for that table is an auto-assigned id

[ChrisV]

@bobman0330

• I just learned that there are these things called indexes, which seem like they are probably useful for this situation;

Yeah, you want an index on associated_entity. You probably also want an index on type, and on related_entity in Names. Indexes speed up searches against that column at the cost of slowing down INSERT and UPDATE operations; usually this is a good tradeoff for columns searched more frequently than updated.

@bobman0330

• The ManyToMany relationship is actually a OneorTwoToMany relationships. I could replace the associated_entity field with a first_associated_entity and an option second_associated_entity that would be 1-to-1;

This shouldn’t be necessary to resolve performance problems and is generally a bad idea because it complicates and probably slows down join queries later.

@bobman0330

• The type field on Record is currently a string because I didn’t know all the possible types, but there are really only 500-ish types. I could convert this into an integer field.

It’s generally better to do this if you can.

@bobman0330

• The 20 million records are mostly not relevant to my application, so in practice I want to use the mega database as the source for a smaller database that will only have records of a particular type (call it a million per child database), with additional fields relevant to that type of record. Maybe that will make my performance problems go away? What is the time complexity of WHEREs?

Time complexity depends heavily on design, indexing etc. 20 million records doing joins on integer keys is not that many if indexed. I think you’ll find that there’s no need to split the database up once you have type farmed out to a lookup table and everything indexed.

[blackize]

In mysql some (all?) table alterations result in rewriting the entire table and can be exceptionally slow with large datasets.

Postgres did this when giving a column a default value until pretty recently.

[microbet]

var groups = [{id: 1, name: “admins”}, {id: 2, name: “group1”}, {id:3, name: “group2”}];
var isThere = false;
groups.forEach(function(element) {
if (element.name === ‘group1’) {
isThere = true;
}
});
alert(isThere)

[ChrisV]

I’m not a Javascript expert but I think this should work:

if(groups.map(x => x.name).includes(“group1”)) { … }

[microbet]

That works too

[suzzer99]

Use const instead of var.

You can use filter or some (looks like some is fine unless you need to support IE 8 or worse).

const isInGroup = groups.some(g => g.name === 'group1');

const isInGroup = (groups.filter(g => g.name === 'group1').length > 0);

[bobman0330]

Other options:

groups.filter( group => group.name ==="group1).length

groups.reduce( (alreadyMatched, group) => alreadyMatched || group.name === "group1")

The last one seems like it should be mildly more efficient since it doesn’t need to create a new data structure and it can short-circuit further lookups after it finds a match. Not nearly as readable as the map solution though.

[grue]

This literally what array some was made for.

[microbet]

Cool. Never heard of that.

if (groups.some(x => x.name === ‘group1’)) {…}

[suzzer99]

The other thing you can do if you need to check the user groups a lot, and you don’t care about group id, is flatten out the groups array into a hashmap.

const reducer = (accumulator, currentValue) => { accumulator[currentValue.name] = true; return accumulator; } ;

currentUser.groups = currentUser.groups.reduce(reducer, {}); 

Then through the rest of the code you can just say (if user.groups.admins). You could even use id as the hashmap value instead of true if you need it somehow.

[ChrisV]

https://www.nature.com/articles/s41598-020-60661-8

Speed of learning Python is better predicted by natural language aptitude than mathematical aptitude.

[suzzer99]

They should do the exact same study with a functional language and machine language.

[simplicitus]

Haven’t read this thread but did some programming in college. Dad was a programmer (started at like 45 y/o with no degree in late 70s) and brother is a programmer at Raytheon.

Anyway, got sucked into a 110 page PDF “book”, The PhD Grind, last night where a Stanford CS PhD and current prof describes his progress through the degree and the politics and trials and tribulations of completing a CS PhD. Pretty engrossing read.

The book is downloadable from his webpage. http://pgbovine.net/PhD-memoir.htm

[grue]

If a user has their system preferences set to dark theme, is it reasonable to expect that an application they’ve used for some time that only has a light theme be automatically set to dark after its upgraded with that capability?

[kerowo]

A better option would providing the user with a dialogue box telling them about the new feature and presenting them with an option to turn it on immediately. Not a fan of apps doing stuff I didn’t ask for.

[microbet]

How did they have dark before the upgrade?

Dark was available, then not, then it came back? The application had different parts with different themes?