Tesla Full Self Driving, comma 3X + openpilot – how do AI-powered self-driving cars currently work?

Tesla Full Self Driving, comma 3X + openpilot – how do AI-powered self-driving cars currently work?

Self-driving cars have been a topic of intense discussion and development for several years now. While many companies focus on a range of sensors like LIDAR, radar, and ultrasonic sensors, there are companies that insist on a vision-only approach.

What is the LIDAR approach?

Traditionally, autonomous vehicles have been equipped with an array of expensive sensors. LIDAR, short for Light Detection and Ranging, uses lasers to create a 3D map of the car’s surroundings. Radar helps in detecting objects’ speed and distance, and ultrasonic sensors aid in close-range detection. These systems often require a lot of computational power and can significantly drive up the cost of the vehicle.

What is the vision approach?

A vision-only approach is relies mainly on cameras, coupled with advanced machine learning algorithms, to give the vehicle its “sense” of direction. Cameras serve as the eyes of the car, capturing images that are then processed in real-time by the vehicle’s onboard computer. This setup significantly reduces costs and computational requirements.

Tesla is famous for its efforts to crack self-driving using a vision-only approach, but there is also an open source startup called comma AI that has a vision-based system called openpilot, as well as sells the hardware to run openpilot on called comma 3X in its latest iteration.

How does the computer understand how to drive a car?

By now a lot of us are already all familiar with OpenAI’s ChatGPT, a generative AI tool trained on a whole load of data to help it understand the world. So how does an AI know how to drive a car?

The answer is simple – by watching videos on how a car is driven.

Through this process, the AI driving model learns to recognise patterns and distinguish between a pedestrian, a cyclist, another vehicle, or even road signs and signals.

Once the machine learning model understands the environment, it makes real-time driving decisions. Should the car slow down because a pedestrian is crossing? Should it speed up to merge onto a highway? All of these decisions are made based on the continuous input of visual data and the patterns the model has learned.

For a human to get better at driving, it has to have more driving experience. The same goes with the AI model. It improves over time as it is fed more good quality data.

Where does Comma and Tesla get their driving videos?

Comma AI gathers its training data primarily through crowd-sourcing. When users install Comma AI’s hardware into their cars, the devices collect video data as well as other sensor information while driving. This data is then anonymized and used to improve the machine learning model.

This allows the system to be trained on a diverse range of real-world driving conditions, including different types of roads, traffic situations, and even varying weather conditions. Second, the data is naturally updated and expanded, which means the model is continuously refined as more users participate and share their data.

Tesla also collects data from their fleet of vehicles. The video feed from the cameras installed all around your Tesla is being used to train Tesla’s FSD AI.

Of course, it’s not just the video feed that’s needed. That teaches the car how the car should behave, but it doesn’t teach the car how to drive, as in how to control the car. Driving videos are collected along with associated data like steering angles, acceleration, and braking input from human drivers. These serve as the “training examples.”

For example, based on a human driver’s acceleration curve, the AI would be able to learn how to comfortable accelerate the car. If not, the driving would be super jerky, with abrupt acceleration and braking, just like a learner driver.

So the AI is like a child that we are teaching how to drive?

Yes. Imagine teaching a child how to recognize a red traffic light and understand what it means. You would show the child many pictures of red lights and say, “When you see this, you stop.” Over time, the child learns to associate the red light with the action of stopping. This process is similar to how machine learning trains a self-driving car.

First, the machine learning system is shown thousands of images of different traffic lights—red, green, yellow—alongside other road conditions. The system is told which color the light is displaying in each image. Over time, the system learns to identify traffic lights and their colors accurately, just like a child would.

Next, the system needs to learn what to do when it sees each color. It is fed more images, but this time, each image also comes with information about what the car did or should do: stop, go, or slow down. So, when it sees a red light in an image, the information that comes with it would be ‘stop.’

After looking at many images, the computer starts getting the hang of it. When it sees a red light, it knows that it’s time to stop. It has learned this through lots and lots of examples, sort of like practicing over and over again.

The same principle applies to recognizing pedestrians. The system is shown many pictures of people walking, standing, or running near the road. It learns that when it sees a shape that looks like a person, it needs to be cautious and prepared to stop.

After all this learning, the system is tested to make sure it understands correctly. It’s shown new pictures it hasn’t seen before to check if it knows what to do. If it passes these tests, it’s ready to be used in a real car to help it drive by itself.

In this way, the machine learning system gets better at understanding what different road signs, traffic lights, and objects like pedestrians mean and what actions it should take in response.

What is the current state of AI driving?

Here are some videos that show what FSD V12 is currently capable of. FSD V12 is a version that has yet to be released to consumers as a beta, but Elon Musk gave a live demo of it on public roads on X live streaming a few days ago.

Below is what Comma AI is capable of.

What will these AI driving systems cost and when will they be available?

Tesla’s FSD V11 beta can be purchased with a Model 3 or Model Y in Malaysia for RM32,000. But note that since the beta doesn’t work in Malaysia, you would essentially be parking your money with Tesla giving them an interest-free loan until it eventually becomes available.

As for Comma AI, you can actually buy the Comma 3X right now and run openpilot on it. Of course it doesn’t support all cars, so you have to check the list of supported cars (over 250+ currently) to see if you can use it. The Hyundai Ioniq 5 is one example of a car available in Malaysia that Comma 3X supports.

This video will show you how the Comma 3X is installed in your car, in case you are curious as to how it works. It’s basically just like installing a dashcam and plugging it into the OBD port.

It will cost you USD1,250 for the Comma 3X and USD200 for a harness specific to a car. International shipping is USD30. That’s under RM7,000 ringgit. Costs have come down quite significantly compared to previous versions of Comma.

Now if you think these prices are expensive, wait til you hear how LIDAR-based systems cost…

Would you spend money on AI driving capabilities for your car? Share your thoughts in the comments.

Looking to sell your car? Sell it with Carro.

10% discount when you renew your car insurance

Compare prices between different insurer providers and use the promo code 'PAULTAN10' when you make your payment to save the most on your car insurance renewal compared to other competing services.

Car Insurance

Harvinder Sidhu

Harvinder Singh Sidhu thinks there's nothing better than Formula 1, not even sliced bread. Having written about cars since 2006, he plunged head first into the industry out of a passion for all things four-wheeled and everything in between. The F1 enthusiast has been following the sport since 1999 and has been keeping up with it since. In between races he keeps himself busy as the host of the Driven motoring show and as our version of the Joker.



  • Proud owner of Comma.ai’s device in Malaysia with my Corolla Altis (not BukaPilot-ciplak version of Comma2). Compared to Tesla Autopilot, Comma is better. With Tesla’s FSD, thats another story. But Comma’s Taco Bell drive looks promising.

    Like or Dislike: Thumb up 6 Thumb down 0
  • Besides Kia EV6, Corolla Altis one of the best supported cars under Comma.

    Like or Dislike: Thumb up 2 Thumb down 0
  • Gpt V7.0 on Sep 03, 2023 at 2:03 pm

    99.9% autopilot won’t works on msia suburbs road too many without road markings or faded lines damaged road kerb missing road sign or even signboard view obtructed by tree shrubs and traffic light pole or street light pole.

    Like or Dislike: Thumb up 3 Thumb down 3
    • I am not sure about autopilot. But for Comma’s highway driving along PLUS, LPT1, 2, I can say more than 90% of my drive were hands and foot-free. At night, it even performed better. Got highway portion that only have dotted line (recently re-paved), no problem. It can also differentiate old and new surface and use the border between them as the line.

      The only reason usually I disengage are due to Malaysian like to drive super slow on 110km/h highway. I am talking abt 60-90 km/h range.

      Like or Dislike: Thumb up 4 Thumb down 0
      • Ganeson on Sep 04, 2023 at 2:29 pm

        Can you share more details about the system you’ve installed and where to get it from? I would like to check it out.

        Like or Dislike: Thumb up 0 Thumb down 0
        • Hi Ganeson,
          I am using the comma.ai’s previous hardware, called Comma2. Bought from USA. There is a similar system by kommu.ai, by Malaysian. Originally they didnt acknowledge that their software is based on comma.ai’s, but now they do. Kommu.ai is cheaper.

          Comma.ai now have newer and better hardware called Comma3x (this is what the PaulTan’s author mentioned). They just launch navigation based OpenPilot.

          Take note:
          Website: comma.ai
          Software: OpenPilot
          Hardware: Comma2, Comma3, Comma3x

          Like or Dislike: Thumb up 3 Thumb down 0
  • KingKong on Sep 03, 2023 at 3:48 pm

    Anyone knows how to avoid paying import tax to our custom when buying stuff like this from oversea? Legal way of course (e.g. declaring it as part of research in a university etc.)

    Like or Dislike: Thumb up 0 Thumb down 0
  • RichardWong on Sep 03, 2023 at 4:08 pm

    To those who are interested in comparing OpenPilot and Tesla’s AutoPilot (not FSD ya), Google “openpilot vs autopilot consumer reports”, there is a PDF article from consumerreports.org.

    Like or Dislike: Thumb up 1 Thumb down 0
  • self-driving on Sep 03, 2023 at 8:30 pm

    just drive your own car that’s the fun of owning a car, or just get a driver.

    Like or Dislike: Thumb up 2 Thumb down 6
    • It is not fun for all people if the duration is long. I am talking maybe about >6 hours. This is where system like OpenPilot helps. If u want to hire a driver, go on. Its your money. Commenting negatively on an article, is very like Lowyat forum. What can u achieved? U want everyone to be stressed driving? I for example really enjoy my balik kampung journey with OpenPilot. I can eat freely, my hands, foot relaxed, I can look around. Of course no sleeping. Yeah, having a supir might be suitable to those who can get easily sleepy while driving.

      Like or Dislike: Thumb up 0 Thumb down 0
  • Lidar is still the best. This is why you see dedicated self driving cars using lidar instead of camera.

    Like or Dislike: Thumb up 0 Thumb down 0
    • Kinte on Sep 05, 2023 at 9:23 am

      Noooo! Major player like Tesla no longer used Lidar. Dedicated? Are you talking about Waymo? Are they dedicated? How much profit that they got? To me, human is the best driver. We dont need rotating head like Lidar to drive. Even 1-eyed human can drive better than most Malaysian.

      Like or Dislike: Thumb up 0 Thumb down 0
  • I wonder why all vehicles on public road does not have some sort of transceivers installed by default so that they can send and receive signals that flags the location and status of all nearby vehicles. This would probably make traffic/accident management and self-driving tech much more reliable. All the tech are already there, mostly used in logistics sector.

    Privacy rights? I mean, most cars are already on public roads. Security issues perhaps? Your personal mobile phone carries way higher security risk.

    Like or Dislike: Thumb up 1 Thumb down 0
    • Paul Tan on Sep 20, 2023 at 11:39 am

      this is called V2X, where cars can share their sensor data and also receive sensor data from other stuff on the street like traffic lights. i believe japan is one country that they are really trying to implement something like this

      Like or Dislike: Thumb up 0 Thumb down 0

Add a comment