I have been working on Phase 2 of the IDB project. During this phase, we are expected to convert our static website from Phase 1 into a dynamic website. We thought this would be simple since we hd already made our Phase 1 website dynamic. The difference was that while we had manually added 3 rows to each table in the first phase, now, we would need close to 1000 rows. We would need to use RESTful APIs to fill our database. Our problem started with Crunchbase, our data source, limiting the amount of data we could get using their REST API. This is expected since this data is extremely valuable to them. However, for us, we would need a lot more data. For this reason, we had to find a workaround and this was through the use of an older snapshot that they provide for free. Merging the current REST API data with this older data would give us what we need. The biggest difficulty we have been having is integrating the data from the old snapshot. It is much harder to integrate with our table since it comes in the form of large files. Although we got the data from the REST API in a couple of hours, we have spent far longer on the snapshot data and there is still a lot of work left. Adding to this, snapshot does not have data specific to schools, one of our models, since at the time, Crunchbase did not have schools as a model. We will need to use an additional API to get this information.

The next difficulty we have been having this week has to do with Amazon's RDS (Relational Database Service) and our EC2 instance. It has been giving us frequent issues and everytime the database reboots, the website goes down. Since we made our website dynamic in the first phase, our database issues were especially problematic. Our hope is that the graders did not get internal server errors when they were accessing the website or if they did, they checked back in a couple of minutes so that the database would be back up.

This week, we also had a talk by Atlassian. They talked about all of their technologies and some of the issues they encountered. Specifically, they talked about using spikes to solve an optimization issue in HipChat. Having used JIRA and Trello, it was interesting to hear about the Atlassian work environment directly from the engineers.

Right now, the biggest issue is getting all of the data into the database. We are hoping to get this done by Monday.

We have major front-end changes that need to be deployed to our live website. We'll be changing some template files to display all cards. When we do this, one problem we anticipate is a slow front-end sort and filter. If this is indeed the case, we might have to move this to the back-end.

I have been enjoying class over the last week. We have continued talking about relational algebra in class and I like using current technologies in our projects outside of class. The last two weeks have been busy but certainly quite fun as well.

Today is April Fools' Day and that means a lot of Google pranks. You have probably seen Google Gnome and Pac-Maps already but there have been so many more pranks. Check out this compilation. Next, for APIs that do not have good documentation, try using JSONLint to format returned responses and figure out what to do with your data. It can handle large inputs although it can take some time.