We are going to use Python and OpenCV to play the dinosaur game. Here I’m using Windows, and my webcam for the camera feed, but you can use most of the code for Linux and MacOS as well.

We are basically going to play the Chrome Dinosaur game using hand movements from the camera feed.

In this project, you need to detect hand gestures. After detecting the gesture, we’ll assign commands to them. You can even play games with multiple commands using hand gesture recognition.

How gesture recognition works:

Install the Pyautogui library – it helps to control the mouse and keyboard without any user interaction,
Convert it into HSV,
Find contours,
Assign command at any value – below we used 5 (from hand) to jump.

Full code to play the dino game with hand gestures:

import numpy as np
import cv2
import math
import pyautogui

# Open Camera
capture = cv2.VideoCapture(0)

while capture.isOpened():

    # Capture frames from the camera
    ret, frame = capture.read()

    # Get hand data from the rectangle sub window
    cv2.rectangle(frame, (100, 100), (300, 300), (0, 255, 0), 0)
    crop_image = frame[100:300, 100:300]

    # Apply Gaussian blur
    blur = cv2.GaussianBlur(crop_image, (3, 3), 0)

    # Change color-space from BGR -> HSV
    hsv = cv2.cvtColor(blur, cv2.COLOR_BGR2HSV)

    # Create a binary image with where white will be skin colors and rest is black
    mask2 = cv2.inRange(hsv, np.array([2, 0, 0]), np.array([20, 255, 255]))

    # Kernel for morphological transformation
    kernel = np.ones((5, 5))

    # Apply morphological transformations to filter out the background noise
    dilation = cv2.dilate(mask2, kernel, iterations=1)
    erosion = cv2.erode(dilation, kernel, iterations=1)

    # Apply Gaussian Blur and Threshold
    filtered = cv2.GaussianBlur(erosion, (3, 3), 0)
    ret, thresh = cv2.threshold(filtered, 127, 255, 0)

    # Find contours
    contours, hierarchy = cv2.findContours(thresh, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)

    try:
        # Find contour with maximum area
        contour = max(contours, key=lambda x: cv2.contourArea(x))

        # Create bounding rectangle around the contour
        x, y, w, h = cv2.boundingRect(contour)
        cv2.rectangle(crop_image, (x, y), (x + w, y + h), (0, 0, 255), 0)

        # Find convex hull
        hull = cv2.convexHull(contour)

        # Draw contour
        drawing = np.zeros(crop_image.shape, np.uint8)
        cv2.drawContours(drawing, [contour], -1, (0, 255, 0), 0)
        cv2.drawContours(drawing, [hull], -1, (0, 0, 255), 0)

        # Find convexity defects
        hull = cv2.convexHull(contour, returnPoints=False)
        defects = cv2.convexityDefects(contour, hull)

        # Use cosine rule to find angle of the far point from the start and end point
        # i.e. the convex points (the finger tips) for all defects
        count_defects = 0
        for i in range(defects.shape[0]):
            s, e, f, d = defects[i, 0]
            start = tuple(contour[s][0])
            end = tuple(contour[e][0])
            far = tuple(contour[f][0])

            a = math.sqrt((end[0] - start[0]) ** 2 + (end[1] - start[1]) ** 2)
            b = math.sqrt((far[0] - start[0]) ** 2 + (far[1] - start[1]) ** 2)
            c = math.sqrt((end[0] - far[0]) ** 2 + (end[1] - far[1]) ** 2)
            angle = (math.acos((b ** 2 + c ** 2 - a ** 2) / (2 * b * c)) * 180) / 3.14

            # if angle >= 90 draw a circle at the far point
            if angle <= 90:
                count_defects += 1
                cv2.circle(crop_image, far, 1, [0, 0, 255], -1)

            cv2.line(crop_image, start, end, [0, 255, 0], 2)

        # Press SPACE if condition is match
        if count_defects >= 4:
            pyautogui.press('space')
            cv2.putText(frame, "JUMP", (115, 80), cv2.FONT_HERSHEY_SIMPLEX, 2, 2, 2)

    except Exception as e:
        pass  # Handle exceptions if any

    # Display the frame
    cv2.imshow("Hand Gesture Recognition", frame)

    # Close the camera if 'q' key is pressed
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

# Release the camera and close all windows
capture.release()
cv2.destroyAllWindows()

How to run the above code

once you have run your file from the terminal that is command prompt

Run the program and switch over to chrome and open chrome://dino

Keep your hand within the rectangle as you start. Move your hand to make dino jump.

Explanation of each line of code above:

1. import numpy as np: Imports the NumPy library and gives it the alias ‘np’ for easier use in the code.
2. import cv2: Imports the OpenCV library, which is used for computer vision tasks like image and video processing.
3. import math: Imports the math module, providing mathematical functions like square root and trigonometric calculations.
4. import pyautogui: Imports the PyAutoGUI library, which allows control of the mouse and keyboard for automation tasks.
5. capture = cv2.VideoCapture(0): Initializes a video capture object to access the camera (index 0 is typically the default camera).
6. while capture.isOpened():: Starts a loop that runs as long as the camera is open and accessible.
7. ret, frame = capture.read(): Reads a frame from the camera and stores it in the ‘frame’ variable.
8. cv2.rectangle(frame, (100, 100), (300, 300), (0, 255, 0), 0): Draws a rectangle on the frame from coordinates (100, 100) to (300, 300) with a green color.
9. crop_image = frame[100:300, 100:300]: Extracts a region of interest (ROI) from the frame based on the rectangle’s coordinates.
10. blur = cv2.GaussianBlur(crop_image, (3, 3), 0): Applies Gaussian blur to the cropped image to reduce noise and smooth edges.
11. hsv = cv2.cvtColor(blur, cv2.COLOR_BGR2HSV): Converts the blurred image from BGR color space to HSV color space.
12. mask2 = cv2.inRange(hsv, np.array([2, 0, 0]), np.array([20, 255, 255])): Creates a binary mask to isolate skin colors within a specified range in the HSV color space.
13. kernel = np.ones((5, 5)): Defines a kernel for morphological operations like dilation and erosion.
14. dilation = cv2.dilate(mask2, kernel, iterations=1): Dilates the binary mask to fill gaps and smooth edges.
15. erosion = cv2.erode(dilation, kernel, iterations=1): Erodes the dilated mask to remove small noise and refine the shape.
16. filtered = cv2.GaussianBlur(erosion, (3, 3), 0): Applies Gaussian blur again to the processed image.
17. ret, thresh = cv2.threshold(filtered, 127, 255, 0): Applies a binary threshold to create a binary image based on intensity values.
18. contours, hierarchy = cv2.findContours(thresh, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE): Finds contours in the binary image.
19. contour = max(contours, key=lambda x: cv2.contourArea(x)): Finds the largest contour based on area.
20. x, y, w, h = cv2.boundingRect(contour): Computes the bounding rectangle around the largest contour.
21. cv2.rectangle(crop_image, (x, y), (x + w, y + h), (0, 0, 255), 0): Draws a rectangle around the detected hand region.
22. hull = cv2.convexHull(contour): Finds the convex hull of the hand region.
23. cv2.drawContours(drawing, [contour], -1, (0, 255, 0), 0): Draws the contour of the hand region on a blank image.
24. cv2.drawContours(drawing, [hull], -1, (0, 0, 255), 0): Draws the convex hull of the hand region on the same blank image.
25. hull = cv2.convexHull(contour, returnPoints=False): Computes the convex hull indices.
26. defects = cv2.convexityDefects(contour, hull): Finds the convexity defects in the hand region.
27. for i in range(defects.shape[0]):: Loops through the convexity defects.
28. Inside the loop, it calculates the angle between fingers using the cosine rule.
29. If the angle is less than or equal to 90, it counts it as a defect and draws a circle at the fingertip.
30. cv2.imshow("Hand Gesture Recognition", frame): Displays the processed frame with hand gestures recognized.
31. if cv2.waitKey(1) & 0xFF == ord('q'):: Waits for a ‘q’ key press to break out of the loop.
32. capture.release(): Releases the camera resources.
33. cv2.destroyAllWindows(): Closes all OpenCV windows.

Conclusion:

In conclusion, using OpenCV to play Chrome’s Dinosaur Game offers a fascinating application of computer vision. By capturing the game screen, detecting obstacles, and simulating key presses, we can automate gameplay. This project showcases the versatility of OpenCV in solving real-world problems and highlights the potential for creative and practical applications in gaming and beyond.