In fall 2025, the East Bay Municipal Utility District (EBMUD) will launch a new Artificial Intelligence (AI) program to collect data on the number and condition of fish in the Lower Mokelumne River. It is developing the program in partnership with Cramer Fish Sciences (CFS), a Portland, Oregon-based company.
Video footage of salmon mapped in a digital frame. The red and blue dots show the coordinates of the salmon’s positions. Image: EBMUD
The Mokelumne River is a 95 mile long waterway that flows west from the central Sierra Nevada into the Central Valley. The Lower Mokelumne River is the portion below Camanche Dam, which is about 20 miles from East Lodi. The species of concern in the Lower Mokelumne River include Chinook salmon and steelhead trout. The health and number of the fish are indicators of the health of the river overall. During the 2024-2025 season, there was a record number of fall-run Chinook salmon, with 35,450 counted.
CSPA has been a strong advocate for the protection of Lower Mokelumne River fisheries for almost four decades. Since 2014, CSPA has been the only environmental non-profit that participates in EBMUD’s partnership committee and technical advisory committee. These committees assist EBMUD in reviewing and making decisions about the management of the now-robust Chinook salmon and steelhead trout populations in the Lower Mokelumne River.
How the work got started
The effort involving AI is a modification of EBMUD’s ongoing effort to study the number, health, and behaviors of Chinook salmon, steelhead trout, and other fish in the Lower Mokelumne River. EBMUD started this work in 1990. It is doing the work at a monitoring station in the Woodbridge Irrigation District Dam (WIDD) fish ladder. This dam was rebuilt in 2005. WIDD is downstream about 24 river miles from the Mokelumne River Hatchery in Clements. As of 2025, the norm is for the hatchery to raise 9 million juvenile Chinook salmon and 250,000 steelhead trout per year.
Woodbridge Irrigation District Dam (WIDD) in Lodi. Image: EBMUD
The Lower Mokelumne River and fish-related structures. Image: EBMUD
How EBMUD’s video system works
EBMUD’s monitoring system includes a fish monitoring window and an Internet Protocol (IP) camera. The migrating fish travel up a permanent fish ladder on the east bank of the river. They pass through the window, where the camera is pointed at them.
An IP camera is a digital video camera similar to a closed-circuit television (CCTV) camera. The difference is an IP camera requires only a local area network to operate. This IP camera is backed up by a Network Video Recorder (NVR), a type of surveillance system. The IP camera codes and processes video data, and then streams it to the NVR. The NVR stores the data and allows it to be viewed remotely. Going forward, EBMUD expects to need a more powerful computer to run the AI program.
EBMUD’s system captures videos of fish traveling up the fish ladder.
Image: EBMUD
How the AI program works
Andrew Veary, image analyst for Cramer Fish Science, wrote and tested the AI program. He wrote the program primarily in a programming language called Python. The program consists of four different deep learning models to capture and translate various visual data about the fish. There are:
- a detection model, to detect each fish in an image frame
- a segmentation model, to segment each fish within the frame
- an automated measurement model, to determine the length and height of each fish and
- a tracking model, to determine the movement of each fish as it travels past the window.
The program uses a depth estimation model to predict the distance of the fish from the camera. This means a fish further back will be represented by different values than a fish closer to the camera.
A depth map tool to determine fork length. The image depth changes depending on how close the fish is to the window/camera. Image: EBMUD
The EBMUD team created a set of flat wood fish to have a known measurement against which they could compare actual fish. EBMUD ran the wooden fish through the WIDD fish ladder at various depths and distances to simulate actual fish passages.
Wood fish cutouts to compare with images of actual fish
Image: EBMUD
“The key points of the fish that we measure (for video monitoring) include the snout, the eye, part of the gill, the dorsal fin (the tall fin in the middle of the fish’s back), the caudal fin (the tail fin), and the peduncle, the narrow part behind the dorsal fin and the caudal fin. These data indicate what species a fish is,” said Matt Saldate, fisheries and wildlife biologist II with EBMUD’s Fisheries & Wildlife Division
As a note, for EBMUD’s normal, hands-on monitoring, staff also record fish species, sex, lifestage, tags and marks, passage time, passage direction, physical anomalies, and fork length, the length of a fish from the tip of the snout to the end of the middle caudal fin rays.
EBMUD staff have assisted Veary since summer 2024 by providing him with annotations for the images of fish they have been analyzing. The staff notes share how people have been counting, measuring, differentiating and noting. The notes have helped Veary refine the program and help it accomplish more tasks. Veary will continue to work on the program until midsummer.
The nitty gritty of monitoring fish
Each year, fall-run Chinook salmon video monitoring operations start on August 1 and continue until the end of January, although monitoring of the WIDD fish ladder continues year-round. The AI system is meant to replicate everything EBMUD’s human reviewers currently do. The tasks include:
- counting the number of each species of fish passing up the fish ladder
- differentiating between male and female fish
- fish measurements and
- determining whether the adipose fin of the fish has been clipped, meaning the fish came from the fish hatchery. The adipose fin is the small back fin of the fish before the tail.
Information EBMUD can observe with a video monitoring system.
Image: EBMUD
From snout to gill, a steelhead trout looks similar to a Chinook salmon. The two species have a very different anal fin, a fin on the underside of the fish.. The measurements can also estimate a fish’s age. Generally, for Chinook salmon, a grilse, or two-year old, is 650 mm and smaller. Fish that are 700 mm or above are typically adults, or over three years old.
So far, EBMUD’s staff have put an emphasis on sharing annotations about how they review images of Chinook salmon and steelhead trout. Going forward, they hope to make more annotations about how they review other species of fish, including black bass, striped bass, Sacramento pikeminnow, Sacramento suckers, and Pacific lamprey. EBMUD terms these species “incidental species.”
Different fish call the Lower Mokelumne River home. Image: EBMUD
A confusion matrix shows how well the system is classifying fish so far.
The positives on the right point out fish that are not entirely in the screen. Image: EBMUD
Earlier in his life, Veary observed commercial fishermen on the East Coast. This helped him integrate Machine Learning (ML) techniques with the fish models.
“You have to run the models simultaneously because you don’t know which fish will come up the ladder at any given time. The AI program must track the different model predictions all at once. The program requires a lot of processing power,” said Veary.
Right now, the programming goes smoothly because of the lack of noise in the system. The term “noise” refers to unwanted data in a frame when a system is detecting or analyzing the frame. In the fish monitoring system, debris and plant material floating through the frame could create noise that interferes with EBMUD’s analysis.
“In the future, we may need to use more sophisticated techniques to monitor smaller, faster fish,” said Veary.
When one or more fish move back and forth rapidly, it can be difficult to monitor their condition and ultimate destination.
“If you have accurate predictions every frame, and you know all the fish will go down the fish ladder, it’s easy to predict a fish’s location in the following frames. (This simplifies) the tracking process,” said Veary.
It helps to remember that each image is a multidimensional array. Every pixel in the image corresponds to an “x” coordinate on an “x,y” graph of the image.
“The program is also great to monitor species like otters that are in there with fish, but look nothing like a fish. The AI program should make it easier to monitor and research them too,” said Veary.
Fixing mistakes
Validating Veary’s work requires EBMUD staff to do spot check evaluations. This indicates how the AI program compares with annotations by trained EBMUD staff.
Veary does not expect the AI program to do its work with 100 percent accuracy at this point.
“The EBMUD staff are flagging fish or other animals like turtles that could be potentially misleading or miscounted. They run videos of the “problem” organisms and take screenshots. Then I rewrite the code to fix the model until it’s perfect,” said Veary.
Without feedback, an AI program can underperform. It is extremely costly to go back and correctly train the AI’s data set.
“I am learning a lot from doing this work. I want to do more projects like this because fish are easy to work with,” said Veary.
Since Veary started work, he has done everything remotely. This summer, he plans to visit EBMUD staff who have been helping him.
“I will look at the camera and determine what could change. Maybe we could alter the placement of the camera. If we narrow the field of view it captures, the fish will have less choice as to proximity. They can’t get far away, so we could potentially see them better,” said Veary.
He also looks forward to building relationships with the staff who have helped him.
The ultimate goal is for EBMUD and partner agencies to set a course to improve the habitat, modify hatchery practices, better coordinate Delta operations and continue the study on the Mokelumne River.
“We want to see the number of Chinook salmon, steelhead trout, and other native fish species in the Lower Mokelumne River continue to increase. We hope what we do here will help restore river systems in the Central Valley. These actions could also offer us ideas about how to manage these rivers to minimize the impacts of climate change,” said Saldate.