How To Create A Computer Vision Dataset From Video in R

I wanted to write a quick article about creating image datasets from video for computer vision.  Here we'll be taking a video that I took on my phone and create training and validation datasets from the video in R.  My hope is that someone who is new to playing with computer vision stumbles on this article and that I'm able to save this person some time and extra googling.  I get giddy when I find a blog article that does exactly what I want and is simple to understand, I'm just trying to pay it forward. The project I'm working on is written in python, so unfortunately I won't be helping you go end-to-end here, unless you're looking to continue in python. To create the dataset, I used the av library in R.  The av library in R makes it crazy simple to split a video you take on your phone into a bunch of images and save them in a folder.  Once you have that, you'll of course need to take a random sample of files to place in a training dataset folder you'll create, and then you'll want to place the remaining images in a validation dataset folder.  Easy peasy. I did not attempt to do anything fancy, I'm hoping this will feel very friendly.

######################################################################
###  Creating a folder with a bunch of images from video

###  The only library we need for this:
library("av")

### The path where you've saved the video and where you want your images
video_path = "[path to movie]/[your movie].MOV"
path = "[path to new folder]"

###    set your working directory to be where the files are stored
setwd(path)

###  Function that will give you all your frames in a folder
###  First we're just dumping all of the images into a single folder, we'll split test and 
###  validation afterwards
av_video_images(video = video_path, destdir = path, format = "jpg", fps = NULL)

###    How many images are in that folder?  Just checking for context
length(list.files())

Now we have a folder with all of our images.  Next we're going to take a random sample of 70% of the images for our training set.  Then we'll move those files to a training folder.  Get excited to move some files around!

via GIPHY

####################################################################################
#### Now creating the testing and validation sets


###    Now Take a sample of 70% of the images for the training set, we do not want with replacement 
images_training <- sample(list.files(),length(list.files())*.7, replace = FALSE)


####   Create training and validation folders so we have a place to store our photos
####   If the training folder does not exist, create training folder (with dir.create), else tell me it already exists
ifelse(!dir.exists("training"), dir.create("training"), "Folder exists already")
ifelse(!dir.exists("validation"), dir.create("validation"), "Folder exists already")


###   Place training images in the training folder
###   Here we are going to loop through each image and copy the folder from the old path 
###   to the new path (in our training folder)
for (image in images_training) {
     new_place <- file.path(path, "training",image)  ### pointing to the new training file path
     old_place <- file.path(path,image)
     file.copy(from = old_place, to = new_place)
}

Next we're going to remove the training images from their original folder, so that all we'll have left in the original folder is the validation images.  Just gonna do a little cleanup here.  To do this, we'll simply loop through each image, and in each iteration of the loop, we're removing an image.

via GIPHY

for (image in images_training) {
  file.remove(path, image)
  }

###  Double check that the length looks right
length(list.files())


###    Put remaining image files in validation folder
images_validation <- list.files()

for (image in images_validation) {
  new_place <- file.path(path, "validation", image)
  old_place <- file.path(path,image)
  file.copy(from = old_place, to = new_place)
}

####  Remove the validation images from the old folder (this is just cleanup)
####  For is image in the remaining list of files, remove the image.
for (image in list.files()) {
  file.remove(path, image)
}

Now you're all set up to start using these images from a video you've taken yourself!  If you're playing with computer vision, I highly suggest checking out the cometr library.  With just a couple lines of code it'll store a snapshot of your dependencies, code and anything else you need for your model to be reproducible.  This is an absolute life saver when you later run into a bug and you're not sure if it's a problem with your dependencies, etc.  cometr makes it so you'll be able to just check on your last successful run, easily compare with the current code, see what the discrepancy was, and continue on your merry way.  If the libraries for computer vision that you're using integrate with comet, then you'll also get a bunch of metrics and graphics to help you assess your model right out of the box.

From here, you'll want to create bounding boxes for the images.  The easiest way I've found to do this is leveraging the labelImg library in python.  You just pip install the labelImg package and then run labelImg in python and a GUI pops up for creating the bounding boxes. It really can't get much easier than that.  If you happen upon a great way to label the images that doesn't involve python, please let me know.  I would love to suggest something non-python here because this is obviously not a python article. Thanks for reading!  Hope you have the easiest time turning your video into an image dataset for training and validation, and may your object detection models detect all the things.

Originally published athttps://heartbeat.comet-ml.com/ on June 2, 2022.