*Edit 2, Jan 28, 2016: Didn’t update earlier, but the newest version is 1.0.10.
*Edit 1: So I released a new version (1.0.5) of my gem when I realized I forgot about our friends using operating systems other than Mac OS. The gem should have a smarter interface and be able to open the trailer on Mac, Linux/Unix or Windows operating systems. Please download the newest version as of Tuesday, January 19th, 2016, 7:30 pm.
So I released my first Ruby gem recently. I started the project on Saturday, January 16th, 2016 and finished and published it on Monday, January 18th, 2016. It was a great experience. I learned quite a bit and it was a valuable learning experience. Also shoutout to Vincent Alfieri, Diane Cai, and Sara Tibbetts for all the help they provided me during the project. They were always there to offer advice and were very quick to respond. Vincent, thanks for going above and beyond to help a newbie and putting in that suggestion for including the movie trailers. That was a great idea. Sara, thank you for staying up with me past midnight to help while I was doing the project.
The gem is of course a CLI gem that runs with the command ‘imdb_movies’, which opens up a menu like this:
There you can see more information about movies opening this week, coming soon and the ones currently playing. To get more information about a movie simply enter a part of the movie name and you should get basic information about the film.
You can enter ‘y’ or ‘yes’ immediately after and it will open up the trailer in your browser. Otherwise you can continue browsing films until you are done and enter ‘exit/quit’ to leave the program.
The idea for the project came to me because I wanted to make an application that could be enjoyed by a wider audience. I saw Star Wars: The Force Awakens the a few days prior to the project and thought that movies would be a great place to start. Everyone loves movies right? So my mind went immediately to IMDB’s website and I got to work.
I spent the bulk of my time programming on Sunday, with Saturday being split between designing the gem, looking over Avi’s Daily Deals’ video and programming at night. I had the gem functional by 2am on Monday and made the finishing touches by noon. Most of the afternoon on Monday, was spent trying to figure out how to get my gem on Rubygems.org published the way I wanted. I made a lot of mistakes trying to publish a working version and finally published a presentable version, 1.0.3.
The first step in the building process was of course the CLI’s call method. After initializing and greeting the user it had a loop which would ask the user for input until ‘exit’ was given. It would call the Imdbscraper which would scrape the IMDB’s front page for the information on the side bar.
My original scraper got the headers for every section in the sidebar, the links for the see more pages and the title of the movies. I ran into a few problems scraping their front page. I thought it would have been a simple task since I had some experience from scraping a page from this project. I thought that a site as big as IMDB would be well designed and have ids everywhere on its HTML so that they can make quick design changes if they wanted. This was not the case and I ran into the problem of having scraping the TV links as well as their social media. They also used the same class for this button to get tickets & showtimes. I had no trouble sorting through the different headers but had a problem when I tried to get the links for the see more pages and movie titles.
When I was scraping links that were not there I ran into the issue of the NoMethodError. I would be using the attr method to pull the href from a link that did not exist and Nokogiri would scream in pain and not cooperate. I tried to fix this by adding an if statement at the end to only get the link if it was not nil. This did not work either. When I checked with pry this is what I found. For the sections with button and the social media, the css method would return a Nokogiri object which had no anchor that I was searching for. When I printed what was returned, it gave me nil. However, when I did a comparison with nil, it was false. I’m still not completely sure why it was doing that. My first solution to this problem was to only pull the links if that categories header was equal to “Opening This Week”, “Now Playing”, and “Coming Soon”. I changed that later in the process by including a rescue statement in cases of NoMethodError. I completely forgot that rescue existed for these cases for a day and a half.
After making the ImdbScraper I moved on to the Movie class which would scrape the movies page on initialization. The Movie class was a relatively simple build compared to the ImdbScraper. The only real issues I ran into was finding the code which was getting pass the issue with OpenSSL when it scraped Youtube and getting my program to ignore the ad videos that popped up randomly in testing my trailer scraper. I was pulling the trailer for Dirty Grandpa instead of Star Wars about once in 10 tries.
I made a rookie mistake near the beginning by making my CLI call my ImdbScraper to then store links. At first this didn’t seem like a bad idea until I changed the scraper to create and return instances of Movie from the links from the see more pages. It was a horrible design decision in hindsight because it would scrape all 3 see more pages and create a new object for every single movie upon startup. This made the program lag so much and it would take almost 40 seconds on average before it would even get to the menu. After testing that a few times I scratched that idea and made that scraper call the MovieLinksScraper which would scrape the movie links only from the see more pages.
I realized that I would only need to make the Movie instance when the user actually wanted info on a specific movie. Changing this made it simpler. I made a variable called @imdb_main back in my CLI class which would call ImdbScraper and store the category headers, the links to the see more pages (later stored in @nav_links) and the movie links(@movie_links). Making this decision helped reduce the time it took the run the program since I only needed to store about 40 or so links upon startup and create about 4 objects instead of making about 30 Movie objects, each scraping about 10 types of information. The ImdbScraper now sent back an array with 2 arrays: one containing the category header and the link to the see more page, and another containing arrays each with a movie title and the link to that movie’s page.
After making sure that my scrapers worked and that I could make Movie objects properly, the last big part of the gem was the Display class and getting my CLI to look pretty and improving the user experience. The Display class would get created and display different movie titles or movie info based on the information that my input method inside the CLI sent it. This was the easiest part of course. When I finished it was Monday and I went on to publish my gem. In came a whole host of problems stemming from my inexperience.
I edited me ReadMe. I added my information in the gemspec. I included nokogiri inside my Gemfile. I deleted gemspec from inside my Gemfile. It was good. Wait, what was that? I delete what from the where in the who has it what’s it! While trying to get my bin configured and my Gemfile, Rackfile and gemspec correct, I made a lot of mistakes. First one that I recall was deleting gemspec from my Gemfile. Second one was not requiring the nokogiri gem inside my gemspec. Every other mistake resulted from not knowing how to get my gem to run imdb_movies from the bin file. I looked through different guides on how to build and publish gems. I looked at Avi’s repository for Daily Deals on Github and tried to match what he did. I published my failed gem 3 times and tried testing it on my other laptop before I found my mistakes.
I was trying to make my gem work simply from typing ‘imdb_movies’ from the command line, but I couldn’t configure my gem in a way where it directed to the right file in my bin directory. I kept making changes to the required path at the top of the gemspec and the executeables. After making my mistakes, Vincent linked me to this very helpful guide and I eventually fixed my errors. I learned a very valuable lesson that day. The defaults in a lot of gems are set that way for your own good. Only change what you really need to. It is designed to be idiot proof and get you up and running with 4 line changes.
At the end of the project, I had a good laugh when it finally worked. I think a good 35 people downloaded my faulty code before I published a functional version (although I think 15 of those were me). Three days of work and 4 publications later and I have my first Ruby gem open to the public. It was a fun experience. I managed to make an app had a use that people can download and it was very gratifying. I can now get back to a semi-normal schedule since I skipped on meals and sleep during that time. That is all and for anyone reading this, feel free to download the app. Download it with ‘gem install imdb_movies’ and check it out. I hope you like it and feel free to provide any feedback on Github or you can email me at firstname.lastname@example.org.