Programming

[jmakin]

I am only commenting on this from the AWS infra side of me, and not any particular design comments, because you have a better sense of your problem and constraints than I do, and I hope I’m not misunderstanding anything.

I’ll just say right now anything automated that claims it can maintain what you’re describing is lying. If you don’t believe me, for some reason, imagine if you will AWS could do this - why the hell wouldn’t they offer that as a product already? Or anyone else? Because it isn’t possible, yet, and if you ask me, never will be - and while I can take this opportunity to go on an idealistic spiel, that’s because organizational problems eventually become infrastructure problems, and it can go the other way too, and since all organizations are innately different, there’s no general solution. To make matters worse in this space, AWS itself offers 50 billion ways to solve a problem, so you can easily go down a terrible path without knowing or years of painful experience.

@suzzer99

Basically I need to send an email to to everyone who won an item(s) just after the auction has ended. I have the function that sends the emails. But to schedule it to happen when the auction ends, I need to be able to create, update and remove an AWS scheduler programmatically from the auctions app. (I’ll explain why I can’t just use the app to do it below).

I used ChatGPT to get create some basic lambda handler code to programmatically interact with AWS EventBridge. I even used it to explain to me what the roleArn property should be, how to create the role, what permission and trust relationship to assign. I was surprised that it was able to do this. IAM is always so damn confusing to me - what’s the source, what’s the target, should I create a new role just for this, should I used a managed permission set or roll my own, etc.

So kudos to ChatGPT on that. But here’s all the stuff so far that I think still needs a human to figure out and I don’t see how copilot or whatever could ever realistically decide. Also, not coincidentally, these are the kinds of decisions a junior programmer could really go down the wrong path with and make a mess of.

The “AWS” architect way, I think, because I refuse to take that stupid cert, would be to recommend you fire events to your AppSync, which I’ve never heard of til now and am just guessing, to some AWS lambdas that are stitched together through AWS eventbridge configs, which you are right, are super horrible to guess through and understand what’s going on, and AWS docs aren’t notorious for being amazingly helpful. Eventbridge, formerly known as “cloud watch events” (liked that name better because it actually integrates with cloud watch a lot better than anything else due to how it evolved), is a formidable and powerful tool but involves complex permissions, as you noted with IAM, that are easy to fuck up and come without a lot of guard rails. That is why I mention the audit thing, I don’t know your domain and don’t want to, but these things are very easy to mess up in a terrible way, at least from a compliance/security standpoint.

1. Why not just have the app trigger itself to send the emails when the auction is done? It’s because we’re using AWS AppSync, which is really just a collection of lambdas, which are just inert functions that get executed at run time. There’s no running server to run some kind of internal chron job with.

Yea I imagine these are all fired from eventbridge. You could alternatively just set up a lambda on its own on a schedule, if you don’t wanna do it the “AWS” way with events, just set up a web hook that fires on a schedule you control from elsewhere. you’re an adult, you can deal with how you communicate with it and authenticate to it (if needed). You can even define your own custom authorizers with lambdas, if you want to get really technical with API stuff. The problem with event-driven solutions when you are building them in AWS is they are extremely hard to test in development. Which is why out-of-the-box solutions probably seem appealing, but are a nightmare to maintain.

2. How to solve this? My first idea was just to have a simple node server running on the side that could interact with AppSync and trigger the emails to send at a specific time. But that would involve creating some kind of EC2 instance, either through Elastic Beanstalk, or manually, and a bunch of other permissions and communication type stuff I didn’t feel like dealing with. I do everything I can to avoid having servers to babysit.

External server, if you’re more comfortable setting that up, yes! and skip app sync, why not just have it talk directly to the lambda via a web hook? Not sure I see why the app is involved, but if it has to be, just have the app fire the events to a web hook endpoint (super easy to set up) and skip the complexity of app sync!

3. Another solution that might appeal to a junior developer is to have the admin user’s browser trigger the end of auction. But of course that requires that an admin user have their browser open at the right time, and would be a very bad idea imo.

that is terrible

3. So I decided to go with the scheduler, even though it’s scary because there are a ton of moving parts, and sending out these emails to a bunch of people before the auction is over could be very bad. Which means and I need good integration tests that create an auction from whole cloth, have fake users bid on stuff, end the auction and make sure the emails go out (using Pop3 to connect to Gmail test accounts). Luckily I already have all that stuff in place. But good luck having ChatGPT tell you how to set it all up.

one of these parts will break at some point and you won’t know how to set it up. the docs will tell you to check cloud watch logs, but oh wait, your AI advised IAM role didn’t have the permissions for that and you don’t have logs without spending a day how to set that up.

4. How to interact with the scheduler? Apparently AppSync can talk to AWS EventBridge directly. But that spooks me a little because the AppSync functions and resolvers only allow a small subset of Javascript—including not allowing the ‘new’ keyword, which makes dealing with dates and time zones a gigantic PITA. Also the AppSync logs mush everything together for each request, which is getting harder and harder to trace as our app gets more complicated. So instead I have AppSync talk to a lambda that talks to EventBridge. That way if we have some crisis in real time after the auction ends, or during it, I can quickly go to the dedicated lambda logs to see what’s going on.

skip eventbridge just make outbound network calls to a web hook. That probably applies to the rest below, other than to also say, if you’re designing your code around infrastructure that’s a sign your infrastructure design is bad.

5. EventBridge has separate functions for CreateSchedule and UpdateSchedule. It’s not super easy to tell if the schedule has already been created or not. So should I go through extra hoops to detect if a schedule already exists for this auction, or just delete any existing schedule and create a new one? I went with the latter but I haven’t got it working yet. If I run into weirdness (like time lags for deleting, which AWS does for some resources) I may need to revert to the first method. Also trying to delete a non-existent schedule and then catching the error will work, but it’s kind of code smelly imo.

6. Which logic should go into the AppSync function that calls the lambda and which should go into the lambda itself. IE - how generic should I try to make the lambda? Should it be able to schedule anything for any app? Should it be able to schedule anything, but just for the auction app? Should it be able to only schedule the end of auction event for the auction app?

7. I went with the last thing. I’ve found that trying to make things too generic too early just leads to pain down the road, because inevitably you assume a bunch of stuff that you don’t need, and find out you need stuff you didn’t consider.

8. In general, I’ve found it much easier to just make the thing very specific the first time, then worry about sharing code when you have a second use case, and maybe not even then. Instead of DRY (don’t repeat yourself), I prefer WET (write everything twice). Then maybe the 3rd time think about creating some new layer of abstraction or mechanism to share the code/function. This is something that it usually takes junior developers a long time to figure out. Some never seem to and are always trying to write the perfect abstraction, then getting endlessly frustrated when future requirements throw a monkey wrench into their beautiful crystalline design. I’ve found it at least an order of magnitude easier to add a new layer of abstraction later than remove an unneeded one. Your boss finds this out when they ask you for a new feature on a mature app you created and you go into a conniption fit about how hard it will be, or you say no problem, we can do that pretty easily.

Anyway this is just a sliver of the thought process I’ve had on this so far, and I haven’t finished coding it yet. It’s why I don’t worry about Copilot taking my job for the near future at least. I spend about 90% of my dev time thinking about stuff like this and 10% actually typing out code on the keyboard, if that.

Oh yeah, and I did have to spend some extra time down a typical AWS cryptic error message rabbit hole because ChatGPT was adamant I needed to add a ‘Z’ to the end of the timestamp (ISO minus milliseconds), when the Z definitely should not be there. So there’s always that.

[suzzer99]

I don’t think you need locks as long as you only have one redis instance. Redis is single threaded and everything is atomic. If you’re still on node and want to do a node<->redis design session/sanity check, feel free to hit me up.

Does Redis need locking?

It sounds like Redis locking is for using redis as a distributed lock.

[suzzer99]

Thanks. Some good info in there. AppSync is just a collection of lambdas and an API Gateway instance that form a static graphql server. We decided to go with it for this project to see if it would work, and for the most part it does.

For the record I don’t think anyone is claiming to automate all these decisions. I was just pointing out that when my boss goes to reinvent and gets all excited about AI code helpers, in the end it may make that 5-10% of my job I spend actually coding 20% more efficient or something. Not exactly a huge game changer. What it may do is eliminate a ton of junior devs, which will suck when there are far fewer new senior devs being created.

[grue]

yeah I’ll take whatever code review you want to give but uh as usual I’m mostly working on the UI. Here’s a fun one though, right from the start the top level of the repo is /client, /server, /shared, but the client and server have different tsconfigs (trust me, it took me weeks to get this all working). And neither of them like importing from a ts file outside of its rootConfigs.

So I gave up and am now copy pasting every time I change a shared type =/

[suzzer99]

I avoid typescript unless I’m building a shared library that will be used by other apps. It adds some efficiencies, but also takes a lot of the fun out of developing Javascript. I’ve definitely never had to deal with a shared tsconfig.

[bobman0330]

@suzzer99

I do agree having a DB field for AuctionFinished and having everything key off of that is cleaner than what we’re doing now which is keying off the auction EndTime, primarily because the time zones are such a gigantic PITA. I’m going to look at changing that after Christmas. The only downside is worrying about what happens if EndTime and AuctionFinished somehow get out of sync.

In the long run, all the distributed-systems stuff becomes much simpler if every state change runs through Dynamo and every side effect triggers off of a state change in Dynamo. Dynamo gives you consistent updates, and it’s much easier to unit test lots of permutations of current state + event → new state than it is to try to write integration tests for all the weird race conditions and edge cases you’ll have trying to run around cancelling pending schedules.

You can refactor the details, but the basic bones of state-change lambdas that trigger when stuff happens, interact with Dynamo to get and update the current state, then trigger all the side effects, is nearly always the right basic choice. Letting stuff happen without running through Dynamo makes things much more complicated and buggy.

[ChrisV]

I’m writing a pet project (Polymarket rewards farming bot) and writing in Python for the first time. Really liking it. Concise and intuitive and the type hints and linter replaces most of what you lose by ditching static typing. Getting imports/package handling working was an unbelievable nightmare though, don’t understand how it’s so bad in a mature language.

Also I wrote a slightly complicated (for a Python noob) websockets class that used asyncio and whatnot and wasn’t sure I had it right, so I pasted it to ChatGPT and asked for an explanation in English of what it did and any issues. It gave me an incredibly lucid description of what it was intended to do, correctly pointed out four bugs and then… gave me a rewritten version that introduced a bug. Very on-brand.

[jmakin]

Yea +1 on the packaging thing. I’m in dependency hell right now, one of my ongoing projects we decided to use python for gluing basic 3rd party oauth + a little business logic together using python because it really is very easy to write (except the c guy in me struggled a little with the semantics of truthiness in the language), but the packaging is a complete mess to me, especially trying to get it hooked into any mature production automation. Python language maturation to me following along these days seems like they’re starting to learn a lot of lessons languages learned 20+ years ago and it explains some or a lot of the kruft. for example, a language that set out to be about readability ends up with crap like

if __name__ == ‘__main__’

is just extremely silly to me. reading comment threads about the recent GIL decision is a fascinating view into the human condition

[kerowo]

There is nothing intuitive about using white space as meaningful characters if you’ve learned any other languages first. Python is nice but its popularity grew faster than its capabilities did and it is being used for things I doubt it was ever intended to be used for.

You don’t need to use Python’s magic names, but they make some things easier if you do. I’m sure every language has lower level features that increase functionality at the cost of readability by people who don’t usually use that feature of the language. I’d never seen coalesce in SQL until I started looking at some of the ETL stuff we’re using and it took more SQL books to find out what it did than it should have.

[jmakin]

it was just an example of something i find silly but i agree on whitespace, I spend a lot of time designing and inside of obnoxious yaml templates which has even more annoying whitespace semantics, but working in python’s the first thing that ever got me to install a vertical line separator in my vim setup. If it were up to me everything would be written in a mix of bash, c, and php but we live in a society apparently.

[ChrisV]

Trying to google for help about it was the worst. Google “what package manager should I use” and it’s all articles that are like “here are 15 different options!”. Google “how do I get this fucking language to understand what my import statement means” and it’s like “here are 5 ways you might approach doing this!”. DIE.

IDK if it will help you if you’re trying to integrate it into a production environment, but I ended up using UV, which is a pretty new Python package manager, on the theory that if there currently is no dominant package manager in the community, you want to go with the new one that has learnt from the downsides of the others. I thought I was onto a good thing when its Windows installation instructions were just “run this command in PowerShell”. When you run “uv init” in a directory it sets up a virtual environment for you, handles adding everything to .gitignore, etc. Adding packages is just “uv add”, setting everything up again when you git clone into a new location is “uv sync”. Thus far it just works without any sys.path or PYTHONPATH or any of that stupidity.

[ChrisV]

@kerowo

There is nothing intuitive about using white space as meaningful characters if you’ve learned any other languages first.

It has been for me. I hate it when code isn’t well formatted anyway, so using that for syntax suits me. I can’t say I have ever thought my Python experience would be improved by sprinkling curly braces everywhere.

@jmakin

it was just an example of something i find silly but i agree on whitespace, I spend a lot of time designing and inside of obnoxious yaml templates which has even more annoying whitespace semantics, but working in python’s the first thing that ever got me to install a vertical line separator in my vim setup.

I could see it being annoying in a generic editor. I am currently using PyCharm - started off in VS Code but had issues with the debugger - and since it understands the language syntax it pretty much always knows where I want the cursor to be.

[jmakin]

yea it is very annoying, especially on days I have to visually review several hundred to thousands of lines of code, in vim on languages that I work in the most, I can make macros that hop across closing brackets, but whitespace sensitive languages i havent quite figured out yet

[suzzer99]

I’m all for anything that reduces clutter and lets me see more code at once. So I approve of whitespace brackets.

[kerowo]

You white space hippies are only happy with it because you never worked on code on two computers with different number of spaces in their tabs.

[suzzer99]

Always convert tabs to spaces. Never leave tabs in code.

And always use two-space tabs, anything else is ridiculous.

[kerowo]

Sure, you probably code in 4 point font too with the other optical mutants!

[jmakin]

you guys are triggering me

[suzzer99]

Big font is exactly why I don’t want 4-space tabs. I don’t want lines running off the screen or wrapping.

Can we all agree that 8-space tab people should be committed to a mental health facility for their own and others’ safety?

I also cannot stand } else { on the same line. But I know I’m in the minority on that.

[microbet]

I do like the look of Python code, but I may in a really tiny minority in that I don’t really care about whitespace or not, indent or tab or by how much, braces, brackets, semi-colons or any of this very much at all. And, oh my the man hours wasted at my work on the linters; installing, automating, talking about, “fixing” older code, and frustratingly having different linting/formatting tools disagree with each other for things that, to me anyway, make almost no difference.