Building a CRUD application

Leave a comment

Recently I made a new CRUD application. It is basically a Reddit clone, where I tried to recreate most of the functionality of Reddit. The source code can be found on Github. The application has all the basic features of Reddit, including user authentication, subscriptions, and the ability to create new posts, subreddits and comments. In the application you can create private or public subreddits, link posts, and text posts. You can also have mod status over subreddits you create or if a mod adds you.

Originally I thought of making this application as a messaging client with subscriptions to chat rooms but choose to save the idea for another day because of the complexity. The application took a while to build. I had a lot of delays from a few problems I had.

A few takeaways that I had from this project are:

 

  1. Comment everything from the start and end of the block of code. When your code starts to get large and when the complexity starts to get larger, you need to know what you are writing and where the blocks end. When you are tired at midnight, it will be very hard to sort through code when you don’t know where the code ends. It helps when you are debugging the same code days later.
  2. If you use ActiveRecord, join tables are not that useful since ActiveRecord takes care of most of the associations already. You do no need 5 extra join tables for things that are loosely connected. Odds are you might even delete those join tables later.
  3. Plan ahead if you want to go to production mode. If you plan to put your application online, make sure that the place you want to host it offers support for everything you want to use. Do not finish your project, then decide you want to put it on Heroku, only to discover that Heroku does not support your database. And when you try migrating your database from Sqlite to Postgres, you find out that even though both are using ActiveRecord, the code you wrote only works for the Sqlite database and not the Postgres database. The way that my navbar grouped subreddits for display did not play well with a Postgres database. Always write your code with your final product in mind. The environments, configurations and databases your uses for development and production should be the same.

 

I had a lot of problems with ActiveRecord stemming from calling associations for objects that didn’t exist yet or no longer existed. The biggest example would be when I tried to implement the mod features. I would have trouble getting access for other users that did not create the private subreddits. I also had trouble with the mod reassignment, when a mod of a subreddit unsubscribed or deleted their accounts. It would delete certain things but not others and Sinatra would throw errors about not having methods for a nil class after it redirected. But this was useful for learning what ActiveRecords #delete and #destroy methods did. After some experimenting, #delete seems to only remove the object you call it on but not the associations to other objects. If you delete a moderator status for a user, that moderator status still exists for the user. The moderator status is gone in the moderators table but when you have user.moderators in a profile page, the loop will still think the moderator record is there, only it is nil and will cause errors. In order to delete the associations you need to use #destroy. It seems like #delete would be fine with child records but #destroy should be used if it is a parent record.

Oh and there are a few other habits I need to commit to later. One is that there should never be a loop inside your controllers. If there is one you are doing things wrong. Put that inside a model. I ran into this when I was doing the code for mod reassignment. It didn’t make sense to have all the logic for looping through user and reassigning their mod status if there were no more mods, inside the controller. It made things so hard to read. The second was to use partials where ever you can. After some point you start to realize  that you are coding similar things in several html/erb files and that it is unnecessary. There was no point in having a separate file for displaying posts, on the front page and on a specific subreddit. There are always ways the make the building process more efficient and as a programmer I should be committing those to memory.

So that is basically the project. It took much longer than I care to admit but there it is. There is also a video of the demo of potato quality on Youtube if you are interested. Thanks for reading.

The problems with scraping other websites

499 Comments

I tried to use my gem recently to see what movies were coming out this week. Everything broke. It wasn’t displaying the correct movies for the opening week. It couldn’t pull the movies information because IMDB changed EVERYTHING about the structure of it’s html. I was going to change my Movie class to scrap the correct information from the pages, then I realized that the movies had no consistent structure for the html.

Some movies had the old structure but had all the classes renamed. Others had everything changed from the layout to the tags used to the class names. There is no discernible similarity between Jane Got A Gun and Kung Fu Panda 3 as far as I’m concerned. Those html pages looked completely different. It was the same case when I looked at other movies. Some only had minor changes like names. Others changed everything about the structure and names and completely rearranged it all. AGGGHHHH!!!! I was so proud of that application too. It did a simple task and looked elegant.

So this is what happens when you scrap other pages for info and they make huge changes regularly. It has been about a week and a half and my gem is all but useless. Unless I have a way of knowing that every movie will have the same layout and names, I can’t do shit about this. IMDB has the right to do anything they want with their site, but damn don’t they know I have an gem to maintain. This is the problem with scraping big sites; they undergo changes to their sites quite often and you have to constantly keep up with their changes. I am so ticked off right now.

First Blog Post- First Ruby Gem | IMDB Movies

85 Comments

*Edit 2, Jan 28, 2016: Didn’t update earlier, but the newest version is 1.0.10.

*Edit 1: So I released a new version (1.0.5) of my gem when I realized I forgot about our friends using operating systems other than Mac OS. The gem should have a smarter interface and be able to open the trailer on Mac, Linux/Unix or Windows operating systems. Please download the newest version as of Tuesday, January 19th, 2016, 7:30 pm.

So I released my first Ruby gem recently. I started the project on Saturday, January 16th, 2016 and finished and published it on Monday, January 18th, 2016. It was a great experience. I learned quite a bit and it was a valuable learning experience. Also shoutout to Vincent Alfieri, Diane Cai, and Sara Tibbetts for all the help they provided me during the project. They were always there to offer advice and were very quick to respond. Vincent, thanks for going above and beyond to help a newbie and putting in that suggestion for including the movie trailers. That was a great idea. Sara, thank you for staying up with me past midnight to help while I was doing the project.

The gem is of course a CLI gem that runs with the command ‘imdb_movies’, which opens up a menu like this:

Screen Shot 2016-01-19 at 12.19.12 PM

There you can see more information about movies opening this week, coming soon and the ones currently playing. To get more information about a movie simply enter a part of the movie name and you should get basic information about the film.

Screen Shot 2016-01-19 at 12.24.10 PM

You can enter ‘y’ or ‘yes’ immediately after and it will open up the trailer in your browser. Otherwise you can continue browsing films until you are done and enter ‘exit/quit’ to leave the program.

The idea for the project came to me because I wanted to make an application that could be enjoyed by a wider audience. I saw Star Wars: The Force Awakens the a few days prior to the project and thought that movies would be  a great place to start. Everyone loves movies right? So my mind went immediately to IMDB’s website and I got to work.

I spent the bulk of my time programming on Sunday, with Saturday being split between designing the gem, looking over Avi’s Daily Deals’ video and programming at night. I had the gem functional by 2am on Monday and made the finishing touches by noon. Most of the afternoon on Monday, was spent trying to figure out how to get my gem on Rubygems.org published the way I wanted. I made a lot of mistakes trying to publish a working version and finally published a presentable version, 1.0.3.

 

The first step in the building process was of course the CLI’s call method. After initializing and greeting the user it had a loop which would ask the user for input until ‘exit’ was given. It would call the Imdbscraper which would scrape the IMDB’s front page for the information on the side bar.

My original scraper got the headers for every section in the sidebar, the links for the see more pages and the title of the movies. I ran into a few problems scraping their front page. I thought it would have been a simple task since I had some experience from scraping a page from this project. I thought that a site as big as IMDB would be well designed and have ids everywhere on its HTML so that they can make quick design changes if they wanted. This was not the case and I ran into the problem of having scraping the TV links as well as their social media. They also used the same class for this button to get tickets & showtimes. I had no trouble sorting through the different headers but had a problem when I tried to get the links for the see more pages and movie titles.

When I was scraping links that were not there I ran into the issue of the NoMethodError. I would be using the attr method to pull the href from a link that did not exist and Nokogiri would scream in pain and not cooperate. I tried to fix this by adding an if statement at the end to only get the link if it was not nil. This did not work either. When I checked with pry this is what I found. For the sections with button and the social media, the css method would return a Nokogiri object which had no anchor that I was searching for. When I printed what was returned, it gave me nil. However, when I did a comparison with nil, it was false. I’m still not completely sure why it was doing that. My first solution to this problem was to only pull the links if that categories header was equal to “Opening This Week”, “Now Playing”, and “Coming Soon”. I changed that later in the process by including a rescue statement in cases of NoMethodError. I completely forgot that rescue existed for these cases for a day and a half.

After making the ImdbScraper I moved on to the Movie class which would scrape the movies page on initialization. The Movie class was a relatively simple build compared to the ImdbScraper. The only real issues I ran into was finding the code which was getting pass the issue with OpenSSL when it scraped Youtube and getting my program to ignore the ad videos that popped up randomly in testing my trailer scraper. I was pulling the trailer for Dirty Grandpa instead of Star Wars about once in 10 tries.

I made a rookie mistake near the beginning by making my CLI call my ImdbScraper to then store links. At first this didn’t seem like a bad idea until I changed the scraper to create and return instances of Movie from the links from the see more pages. It was a horrible design decision in hindsight because it would scrape all 3 see more pages and create a new object for every single movie upon startup. This made the program lag so much and it would take almost 40 seconds on average before it would even get to the menu. After testing that a few times I scratched that idea and made that scraper call the MovieLinksScraper which would scrape the movie links only from the see more pages.

I realized that I would only need to make the Movie instance when the user actually wanted info on a specific movie. Changing this made it simpler. I made a variable called @imdb_main back in my CLI class which would call ImdbScraper and store the category headers, the links to the see more pages (later stored in @nav_links) and the movie links(@movie_links). Making this decision helped reduce the time it took the run the program since I only needed to store about 40 or so links upon startup and create about 4 objects instead of making about 30 Movie objects, each scraping about 10 types of information. The ImdbScraper now sent back an array with 2 arrays: one containing the category header and the link to the see more page, and another containing arrays each with a movie title and the link to that movie’s page.

After making sure that my scrapers worked and that I could make Movie objects properly, the last big part of the gem was the Display class and getting my CLI to look pretty and improving the user experience. The Display class would get created and display different movie titles or movie info based on the information that my input method inside the CLI sent it. This was the easiest part of course. When I finished it was Monday and I went on to publish my gem. In came a whole host of problems stemming from my inexperience.

I edited me ReadMe. I added my information in the gemspec. I included nokogiri inside my Gemfile. I deleted gemspec from inside my Gemfile. It was good. Wait, what was that? I delete what from the where in the who has it what’s it! While trying to get my bin configured and my Gemfile, Rackfile and gemspec correct, I made a lot of mistakes. First one that I recall was deleting gemspec from my Gemfile. Second one was not requiring the nokogiri gem inside my gemspec. Every other mistake resulted from not knowing how to get my gem to run imdb_movies from the bin file. I looked through different guides on how to build and publish gems. I looked at Avi’s repository for Daily Deals on Github and tried to match what he did. I published my failed gem 3 times and tried testing it on my other laptop before I found my mistakes.

I was trying to make my gem work simply from typing ‘imdb_movies’ from the command line, but I couldn’t configure my gem in a way where it directed to the right file in my bin directory. I kept making changes to the required path at the top of the gemspec and the executeables. After making my mistakes, Vincent linked me to this very helpful guide and I eventually fixed my errors. I learned a very valuable lesson that day. The defaults in a lot of gems are set that way for your own good. Only change what you really need to. It is designed to be idiot proof and get you up and running with 4 line changes.

At the end of the project, I had a good laugh when it finally worked. I think a good 35 people downloaded my faulty code before I published a functional version (although I think 15 of those were me). Three days of work and 4 publications later and I have my first Ruby gem open to the public. It was a fun experience. I managed to make an app had a use that people can download and it was very gratifying. I can now get back to a semi-normal schedule since I skipped on meals and sleep during that time. That is all and for anyone reading this, feel free to download the app. Download it with ‘gem install imdb_movies’ and check it out. I hope you like it and feel free to provide any feedback on Github or you can email me at andy.zheng249@gmail.com.