I think you’re probably right about the circumstances of his departure, and certainly right that this is a bad indicator for where FSD is now. But while I’m sure Karpathy is brilliant and talented, “guy in charge of a mission-critical project that’s stuck in the mud” seems like a prime firing candidate, no?
From what I’ve seen, the CV side of FSD works just fine–the big problems are on the decision-making side. AK gave a talk a couple years ago where, IIRC, he was talking about how they were using the fleet’s cameras to collect hard-to-label images and beam them back to be hand-labeled and used for training. That’s all well and good, but I suspect that, strategically, it’s going to be better to focus on using human drivers to drive supervised learning of the policy network. That’s the reason to go vision-only in the first place (to economically deploy a fleet with production-level FSD sensors so you can start generating training data at scale). Perhaps there was a strategic disagreement about focusing on fine-tuning the vision/sensor-fusion aspects vs locking down the sensors and modelling to focus on the policy side?
One thought I had this morning is that high-profile fatal crashes are probably way more likely to be CV glitches at freeway speeds, while policy weakness is likely to manifest as excessive caution. E.g., if that recent crash in Florida ends up being the fault of FSD, it’s likely that the computer confused the off-ramp for the freeway, then didn’t register the truck the car ran into because it was, in the computer’s mind, parked in the middle of a freeway lane. So it’s a genuinely tough call where to focus.
I hadn’t seen the Florida crash but it looks very much like a CV error, there have been a number of other crashes like that. When you say the CV side works just fine, of course it has gotten good at classifying images. It has to be amazing at that, since that’s the whole way it proposes to work. There are two issues with the CV approach:
A single misclassification at speed, especially of a novel object, could be catastrophic and could lead the car to make fatally wrong assessments of depth. It seems like this is what is going on with Teslas crashing headlong into large solid objects. A LIDAR-based approach is not going to make these mistakes. Even if it can’t figure out what an object is, it is always going to know that it is approaching a solid object.
Classification of single objects and classification of many overlapping, moving objects are very different challenges. I agree that the Reddit guy’s account of extreme hesitancy from FSD driving in Manhattan could be policy weakness, but it could also be the car unable to discern individual pedestrians in a moving crowd on the sidewalk. From what I’ve seen driving in my friend’s Tesla I couldn’t guess at which it is.
Obviously the latter is a big challenge for any FSD system, not just a 100% CV-based system. But that’s the thing, FSD is a huge challenge and I am very skeptical that you are going to succeed by ignoring a huge source of data. Bats navigate their way through very complex 3D spaces (solid objects, other bats, prey) with echolocation, so it’s clearly a viable way to do it. Basically I think that we are faced with too difficult a problem for it not to require all the tools at our disposal.
The thing about Tesla’s claims that CV-based FSD is going just great, guys, is that the survival of the company is predicated on it working. Switching to an approach which includes LIDAR is simply not on the table for them. Like if you knew nothing about autonomous vehicles and I simply gave you the information “people are trying to solve a difficult problem, there are approaches A and B, there’s an existing company that are unable to choose B so they’re going with A while every single other player in the field has opted for B even though it’s more expensive” you can draw the correct conclusion from that, i.e. that A is inferior and likely not going to work.
By the way, I hadn’t seen that Tesla quietly filed with the FCC to add a front-facing radar back into future cars:
Again, I think the conclusions to be drawn from this are pretty obvious.
Mr. Musk’s side became concerned about Twitter’s user numbers after the company announced revisions in its first quarter earnings in April, when it said it had overstated its user base for nearly three years through the end of 2021 due to an error in how it accounted for people linked to multiple accounts. The revision reduced the number of monetizable daily active users by 0.9% for the fourth quarter of last year.
Mr. Musk’s personal foundation gave away $23.6 million in fiscal 2020, the most recent year for which data is available, filings show. That represented roughly 0.02% of his net worth as of the end of that year, according to Forbes rankings.
